[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 434833549 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002258] x2apic enabled [ 0.003022] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.006930] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007016] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009000] LSM: Security Framework initializing [ 0.009042] Yama: becoming mindful. [ 0.010024] SELinux: Initializing. [ 0.011065] *** VALIDATE selinux *** [ 0.019302] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024256] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025129] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027035] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028099] *** VALIDATE tmpfs *** [ 0.030334] *** VALIDATE proc *** [ 0.031176] *** VALIDATE cgroup *** [ 0.032006] *** VALIDATE cgroup2 *** [ 0.033064] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034125] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036023] Spectre V2 : User space: Vulnerable [ 0.037005] Speculative Store Bypass: Vulnerable [ 0.042047] debug: unmapping init [mem 0xffffffff8c859000-0xffffffff8c860fff] [ 0.044927] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045746] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052008] ... event mask: 000000070000000f [ 0.054156] rcu: Hierarchical SRCU implementation. [ 0.056424] smp: Bringing up secondary CPUs ... [ 0.057567] x86: Booting SMP configuration: [ 0.058017] .... node #0, CPUs: #1 #2 #3 [ 0.069089] smp: Brought up 1 node, 4 CPUs [ 0.071010] smpboot: Max logical packages: 1 [ 0.072029] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.153142] node 0 deferred pages initialised in 74ms [ 0.157018] devtmpfs: initialized [ 0.158194] x86/mm: Memory block size: 128MB [ 0.162033] gcov: version magic: 0x41383552 [ 0.168396] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.169147] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.174621] pinctrl core: initialized pinctrl subsystem [ 0.178196] [ 0.179022] ************************************************************* [ 0.182020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185015] ** ** [ 0.188015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.190010] ** ** [ 0.193030] ** This means that this kernel is built to expose internal ** [ 0.195012] ** IOMMU data structures, which may compromise security on ** [ 0.197014] ** your system. ** [ 0.200074] ** ** [ 0.203011] ** If you see this message and you are not debugging the ** [ 0.207015] ** kernel, report this immediately to your vendor! ** [ 0.209009] ** ** [ 0.212014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.215015] ************************************************************* [ 0.219787] NET: Registered protocol family 16 [ 0.221424] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.223043] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.226051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.230196] cpuidle: using governor menu [ 0.232518] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.235518] PCI: Using configuration type 1 for base access [ 0.237218] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.247108] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.248015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.249495] cryptd: max_cpu_qlen set to 1000 [ 0.253230] ACPI: Added _OSI(Module Device) [ 0.255011] ACPI: Added _OSI(Processor Device) [ 0.256010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.259020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.262000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.269340] ACPI: Interpreter enabled [ 0.270064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.271008] ACPI: Using IOAPIC for interrupt routing [ 0.272145] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.276164] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.290133] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.292029] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.295015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.298125] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.304627] acpiphp: Slot [2] registered [ 0.306136] acpiphp: Slot [5] registered [ 0.308153] acpiphp: Slot [6] registered [ 0.310159] acpiphp: Slot [3] registered [ 0.312142] acpiphp: Slot [4] registered [ 0.314136] acpiphp: Slot [7] registered [ 0.316152] acpiphp: Slot [8] registered [ 0.318099] acpiphp: Slot [9] registered [ 0.319091] acpiphp: Slot [10] registered [ 0.321162] acpiphp: Slot [11] registered [ 0.323073] acpiphp: Slot [12] registered [ 0.324075] acpiphp: Slot [13] registered [ 0.325155] acpiphp: Slot [14] registered [ 0.327013] acpiphp: Slot [15] registered [ 0.329089] acpiphp: Slot [16] registered [ 0.330050] acpiphp: Slot [17] registered [ 0.332094] acpiphp: Slot [18] registered [ 0.333130] acpiphp: Slot [19] registered [ 0.335072] acpiphp: Slot [20] registered [ 0.336069] acpiphp: Slot [21] registered [ 0.337068] acpiphp: Slot [22] registered [ 0.338048] acpiphp: Slot [23] registered [ 0.339067] acpiphp: Slot [24] registered [ 0.340071] acpiphp: Slot [25] registered [ 0.341104] acpiphp: Slot [26] registered [ 0.343084] acpiphp: Slot [27] registered [ 0.344089] acpiphp: Slot [28] registered [ 0.346169] acpiphp: Slot [29] registered [ 0.347079] acpiphp: Slot [30] registered [ 0.349112] acpiphp: Slot [31] registered [ 0.351064] PCI host bridge to bus 0000:00 [ 0.353016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.355023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.358023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.361024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.364022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.366020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.368215] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.369979] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.373341] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.378902] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.381567] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.384011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.387014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.389048] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.390667] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.394014] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.396039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.399884] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.403011] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.412015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.418014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.423024] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.428013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.435013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.448109] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.458012] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.464013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.479017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.512025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.531648] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.532872] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.535434] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.540536] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.543223] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.549033] iommu: Default domain type: Passthrough [ 0.552570] SCSI subsystem initialized [ 0.554192] ACPI: bus type USB registered [ 0.558103] usbcore: registered new interface driver usbfs [ 0.563313] usbcore: registered new interface driver hub [ 0.564050] usbcore: registered new device driver usb [ 0.565125] pps_core: LinuxPPS API ver. 1 registered [ 0.567009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.570034] PTP clock support registered [ 0.576124] EDAC MC: Ver: 3.0.0 [ 0.578064] PCI: Using ACPI for IRQ routing [ 0.579758] NetLabel: Initializing [ 0.581008] NetLabel: domain hash size = 128 [ 0.582021] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.585063] NetLabel: unlabeled traffic allowed by default [ 0.592200] vgaarb: loaded [ 0.594343] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.596019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.603721] clocksource: Switched to clocksource kvm-clock [ 0.722212] VFS: Disk quotas dquot_6.6.0 [ 0.723976] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.727339] *** VALIDATE ramfs *** [ 0.731538] *** VALIDATE hugetlbfs *** [ 0.733414] pnp: PnP ACPI init [ 0.735915] pnp: PnP ACPI: found 6 devices [ 0.752104] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.755545] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.758119] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.762294] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.765097] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.767961] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.771415] NET: Registered protocol family 2 [ 0.774100] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.779811] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.783247] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.789120] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.793348] TCP: Hash tables configured (established 65536 bind 65536) [ 0.796344] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.799970] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.802817] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.805590] NET: Registered protocol family 1 [ 0.809143] RPC: Registered named UNIX socket transport module. [ 0.811721] RPC: Registered udp transport module. [ 0.813994] RPC: Registered tcp transport module. [ 0.815852] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.818544] NET: Registered protocol family 44 [ 0.820397] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.822970] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.825379] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.828036] PCI: CLS 0 bytes, default 64 [ 0.830269] Unpacking initramfs... [ 2.676945] debug: unmapping init [mem 0xffff93cb3cc64000-0xffff93cb3ffcffff] [ 2.681939] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.684437] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.687589] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.665824] Initialise system trusted keyrings [ 3.667892] Key type blacklist registered [ 3.670357] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.682973] zbud: loaded [ 3.686334] *** VALIDATE nfs *** [ 3.687607] *** VALIDATE nfs4 *** [ 3.689348] pstore: using deflate compression [ 3.693463] Platform Keyring initialized [ 3.924771] NET: Registered protocol family 38 [ 3.929357] Key type asymmetric registered [ 3.933142] Asymmetric key parser 'x509' registered [ 3.940313] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.944625] io scheduler mq-deadline registered [ 3.947845] io scheduler kyber registered [ 3.951844] io scheduler bfq registered [ 3.954542] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.959441] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.964222] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.968857] ACPI: Power Button [PWRF] [ 3.980318] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.990204] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.005131] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.037677] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.067431] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.075444] Non-volatile memory driver v1.3 [ 4.076835] Linux agpgart interface v0.103 [ 4.106829] virtio_blk virtio1: [vda] 144832 512-byte logical blocks (74.2 MB/70.7 MiB) [ 4.109283] vda: detected capacity change from 0 to 74153984 [ 4.144255] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.147549] vdb: detected capacity change from 0 to 1073741824 [ 4.154502] libphy: Fixed MDIO Bus: probed [ 4.160362] usbcore: registered new interface driver usbserial_generic [ 4.163445] usbserial: USB Serial support registered for generic [ 4.166016] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.170847] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.175263] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.178163] mousedev: PS/2 mouse device common for all mice [ 4.183940] rtc_cmos 00:05: RTC can wake from S4 [ 4.190906] rtc_cmos 00:05: registered as rtc0 [ 4.191397] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.193905] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.193946] intel_pstate: CPU model not supported [ 4.202296] hid: raw HID events driver (C) Jiri Kosina [ 4.212426] usbcore: registered new interface driver usbhid [ 4.214370] usbhid: USB HID core driver [ 4.216213] drop_monitor: Initializing network drop monitor service [ 4.219209] Initializing XFRM netlink socket [ 4.221216] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.222291] NET: Registered protocol family 10 [ 4.230243] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.238482] Segment Routing with IPv6 [ 4.239611] NET: Registered protocol family 17 [ 4.241410] mpls_gso: MPLS GSO support [ 4.248298] RAS: Correctable Errors collector initialized. [ 4.250102] AVX version of gcm_enc/dec engaged. [ 4.251573] AES CTR mode by8 optimization enabled [ 4.329921] sched_clock: Marking stable (4329837429, 0)->(5226932827, -897095398) [ 4.338758] registered taskstats version 1 [ 4.340731] Loading compiled-in X.509 certificates [ 4.343703] zswap: loaded using pool lzo/zbud [ 4.373529] Key type big_key registered [ 4.390334] Key type encrypted registered [ 4.393066] ima: No TPM chip found, activating TPM-bypass! [ 4.395428] ima: Allocated hash algorithm: sha1 [ 4.400186] ima: No architecture policies found [ 4.402562] evm: Initialising EVM extended attributes: [ 4.404643] evm: security.selinux [ 4.406114] evm: security.ima [ 4.407364] evm: security.capability [ 4.408943] evm: HMAC attrs: 0x1 [ 4.411654] rtc_cmos 00:05: setting system clock to 2026-06-26 13:26:16 UTC (1782480376) [ 4.419167] debug: unmapping init [mem 0xffffffff8d803000-0xffffffff8d9fffff] [ 4.436900] debug: unmapping init [mem 0xffffffff8c582000-0xffffffff8c858fff] [ 4.449302] Write protecting the kernel read-only data: 28672k [ 4.463224] debug: unmapping init [mem 0xffffffff8ac03000-0xffffffff8adfffff] [ 4.469828] debug: unmapping init [mem 0xffffffff8b514000-0xffffffff8b5fffff] [ 4.573917] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.584552] systemd[1]: Detected virtualization kvm. [ 4.586804] systemd[1]: Detected architecture x86-64. [ 4.589525] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.625826] systemd[1]: No hostname configured. [ 4.632329] systemd[1]: Set hostname to . [ 4.636769] random: systemd: uninitialized urandom read (16 bytes read) [ 4.646951] systemd[1]: Initializing machine ID from random generator. [ 4.732421] random: ln: uninitialized urandom read (6 bytes read) [ 5.036831] random: systemd: uninitialized urandom read (16 bytes read) [ 5.046747] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.077409] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.103652] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.525719] device-mapper: uevent: version 1.0.3 [ 6.528173] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.614923] virtio_net virtio0 ens2: renamed from eth0 [ 7.806547] scsi host0: ata_piix [ 8.071237] scsi host1: ata_piix [ 8.072848] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 8.076164] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 13.002957] random: crng init done [ 13.011335] random: 7 urandom warning(s) missed due to ratelimiting [ 15.571498] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.125387] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.372379] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.855343] SELinux: Disabled at runtime. [ 20.923359] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.937807] systemd[1]: Detected virtualization kvm. [ 20.956174] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.022769] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.028473] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.035623] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.042449] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.049919] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.059983] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.071556] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... Mounting Huge Pages File System... Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ 22.635085] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 23.328492] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.440195] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.473459] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.899500] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.983964] EDAC sbridge: Ver: 1.1.2 [ 27.673483] Key type dns_resolver registered [ 28.210781] NFS: Registering the id_resolver key type [ 28.214443] Key type id_resolver registered [ 28.216795] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg452-client login: [ 46.931340] hrtimer: interrupt took 5342151 ns [ 117.180319] libcfs: loading out-of-tree module taints kernel. [ 117.381664] Key type ._llcrypt registered [ 117.383210] Key type .llcrypt registered [ 118.960627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 118.997601] alg: No test for adler32 (adler32-zlib) [ 121.258463] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 122.782935] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 124.879176] Key type lgssc registered [ 128.919799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 261.378324] Lustre: Mounted lustre-client [ 266.883231] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 282.830395] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing check_logdir /tmp/testlogs/ [ 287.200540] Lustre: lustre-OST0000-osc-ffff93cb9165d000: disconnect after 23s idle [ 288.989259] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing yml_node [ 293.685708] Lustre: DEBUG MARKER: Client: 2.17.54.95 [ 296.729370] Lustre: DEBUG MARKER: MDS: 2.17.54.95 [ 299.302023] Lustre: DEBUG MARKER: OSS: 2.17.54.95 [ 301.104684] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Jun 26 09:31:11 EDT 2026 [ 318.906873] Lustre: Unmounted lustre-client [ 351.401949] Key type lgssc unregistered [ 351.671040] LNet: 8410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.676311] LNetError: 8410:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 351.697045] LNet: Removed LNI 192.168.204.52@tcp [ 352.926223] Key type .llcrypt unregistered [ 352.927990] Key type ._llcrypt unregistered [ 383.818365] Key type ._llcrypt registered [ 383.824972] Key type .llcrypt registered [ 383.858622] LNet: 8993:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 383.858622] mask usage: [+|-] ... [ 384.027902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 384.051576] alg: No test for adler32 (adler32-zlib) [ 385.018690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 385.114679] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 385.144490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 390.081597] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 390.263066] LNet: 9298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 390.273954] LNetError: 9298:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 390.284987] LNet: Removed LNI 192.168.204.52@tcp [ 390.851407] Key type .llcrypt unregistered [ 390.853245] Key type ._llcrypt unregistered [ 397.994231] Key type ._llcrypt registered [ 397.997580] Key type .llcrypt registered [ 399.450060] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:32:49 (1782480769) [ 399.854847] LNet: 9673:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 399.854847] mask usage: [+|-] ... [ 400.104985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 400.128131] alg: No test for adler32 (adler32-zlib) [ 401.103464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.185861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.208094] LNet: 9711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.225703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.326523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 401.352425] LNet: 9718:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 408.960610] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:32:59 (1782480779) [ 409.804840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.847496] LNet: 10300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.893263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.990539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 420.192551] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:33:08 (1782480788) [ 420.232036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.245424] LNet: 10881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.262585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.336362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 427.777506] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:33:18 (1782480798) [ 427.979020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.992733] LNet: 11471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 428.008569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.063569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 435.049940] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:33:25 (1782480805) [ 435.342619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.361761] LNet: 12056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 435.413580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.538469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 435.991551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 443.991167] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 09:33:34 (1782480814) [ 444.138390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.164284] LNet: 12639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 444.208262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.331861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 444.768211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 445.639575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 453.372544] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 09:33:43 (1782480823) [ 453.500814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.562294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.691045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 461.237451] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 09:33:52 (1782480832) [ 461.448536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.492976] LNet: 13826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 461.510825] LNet: 13826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 461.549647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.646063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 461.711697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 462.055229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 462.094194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 462.460546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 462.514766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 463.022206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 463.049978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 463.525408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 463.559675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 463.830163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 463.863846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 464.262384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 464.307329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 464.643831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 464.670547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 465.093069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 465.143944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 465.458719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 465.486527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 465.839108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 465.883363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 466.388758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 466.426969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 466.875167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 466.950249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 473.747463] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 09:34:04 (1782480844) [ 473.891067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.962454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.100200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 474.165622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 481.562958] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 09:34:12 (1782480852) [ 481.756110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.787510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.841702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 481.876983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 488.793613] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 09:34:19 (1782480859) [ 489.062796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.118483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.169916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 489.201143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 495.736089] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 09:34:26 (1782480866) [ 496.054444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.101020] LNet: 16267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 496.103653] LNet: 16267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 496.128683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.205775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 496.274923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 503.149683] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 09:34:33 (1782480873) [ 503.441373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.496469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.570437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 503.611208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 510.265593] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 09:34:40 (1782480880) [ 510.349117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.407521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.551397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 510.634299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 518.142331] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 09:34:48 (1782480888) [ 518.249634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.294728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.394598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 518.951405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 525.638583] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 09:34:56 (1782480896) [ 525.831106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.887561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.123307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 526.477128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 534.004968] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 09:35:04 (1782480904) [ 534.102917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.162296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.349230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 534.576521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 541.249746] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 09:35:12 (1782480912) [ 541.501736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.532285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.011461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 542.361995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 548.968601] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 09:35:19 (1782480919) [ 549.079522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.132859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.285437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 549.385714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 557.900089] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 09:35:28 (1782480928) [ 557.991623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.066464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.211710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 558.290856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 565.696779] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 09:35:36 (1782480936) [ 565.909982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.940212] LNet: 21707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 565.947332] LNet: 21707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 565.997750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.091485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 572.331600] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 09:35:43 (1782480943) [ 572.584956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.630510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.726303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 580.317178] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 09:35:50 (1782480950) [ 580.427204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.514837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.632442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 581.095971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 588.533844] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 09:35:58 (1782480958) [ 588.654197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.694572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.798671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 588.843230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 596.243363] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 09:36:06 (1782480966) [ 596.479607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.541945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.607380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 596.633622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 603.809838] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 09:36:14 (1782480974) [ 604.031407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.086399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.202685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 604.264461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 612.140766] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 09:36:22 (1782480982) [ 612.249288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.337105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.477702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 612.520152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 619.028226] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 09:36:29 (1782480989) [ 619.165965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.254504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.309755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 619.386398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 619.407253] LNetError: 25823:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 619.458936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 619.497183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 625.364320] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 09:36:36 (1782480996) [ 625.604054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.647457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.713205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Zads1JZj44 [ 625.795774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Zads1JZj44 [ 625.840194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Zads1JZj44 [ 632.382892] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 09:36:42 (1782481002) [ 632.476047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.544966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.623970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 632.688486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 632.776493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 639.811995] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 09:36:50 (1782481010) [ 640.149651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.225370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.341924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 643.506718] LNet: Message forwarding will be enabled in 2s [ 646.623235] LNet: Message forwarding enabled [ 647.085680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 647.127633] LNet: Message forwarding will stop in 2s to 6s [ 651.760903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 652.092527] LNet: Message forwarding disabled [ 653.974803] LNet: Message forwarding will be enabled in 2s [ 657.247123] LNet: Message forwarding enabled [ 661.399502] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 09:37:12 (1782481032) [ 661.555862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.108650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.184328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 669.822324] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 09:37:20 (1782481040) [ 670.028210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.101821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.155489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 670.196281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 670.254947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 670.292977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 670.335366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 670.369243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 670.397937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 670.431846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 670.465284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 670.502937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 670.549762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 670.585700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 670.618156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 670.664451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 670.693376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 670.729469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 670.775719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 670.819768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 670.871111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 670.900436] LNetError: 28894:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 676.946515] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 09:37:27 (1782481047) [ 677.284788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.328517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.436556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 684.169305] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 09:37:34 (1782481054) [ 686.251535] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg452-server [ 688.300941] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 09:37:39 (1782481059) [ 688.692861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.738463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.786569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 688.811531] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 690.429765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[53-57]@tcp [ 690.449512] LNet: 30611:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 690.512028] LNetError: 30400:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.53@tcp is being used as a gateway but routing feature is not turned on [ 690.530440] LNetError: 30400:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.53@tcp has gone from up to down [ 690.542455] LNetError: 30400:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 691.495295] LNetError: 30400:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.56@tcp is being used as a gateway but routing feature is not turned on [ 691.512990] LNetError: 30400:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 692.515296] LNetError: 30400:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.55@tcp is being used as a gateway but routing feature is not turned on [ 692.527983] LNetError: 30400:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 694.573751] LNetError: 30400:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.53@tcp is being used as a gateway but routing feature is not turned on [ 694.593351] LNetError: 30400:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 697.907597] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 09:37:48 (1782481068) [ 698.660575] LNetError: 30400:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.53@tcp is being used as a gateway but routing feature is not turned on [ 698.673164] LNetError: 30400:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 700.087826] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg452-server [ 702.282884] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 09:37:52 (1782481072) [ 702.420816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.446551] LNet: 31545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.453902] LNet: 31545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 702.462961] LNetError: 31545:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 702.498478] LNet: Removed LNI 192.168.204.52@tcp [ 702.522289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.574336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 702.602096] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 703.398715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[53-58/2]@tcp [ 703.426424] LNet: 31664:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 703.433614] LNet: 31664:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 703.502043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[53-58/2]@tcp [ 703.502707] LNetError: 31558:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.57@tcp has gone from up to down [ 703.532757] LNetError: 31558:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 711.731388] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 09:38:02 (1782481082) [ 711.847655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.896973] LNetError: 32251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.922833] LNet: Removed LNI 192.168.204.52@tcp [ 711.971754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.186409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 712.288864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.356496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.439730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 712.562807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.614757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.698396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 712.794449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.869995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.012962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.129080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.175608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.261928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 713.351614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.395186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.451894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.533584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.574113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.661077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 713.778454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.834715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.951618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 714.031995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.063631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.138971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 714.239942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.303279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.428276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 714.568339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.619739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.714211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 714.790363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.878866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.981346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 722.898961] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 09:38:13 (1782481093) [ 724.682691] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg452-server [ 726.755290] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 09:38:17 (1782481097) [ 729.000224] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg452-server [ 731.029814] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 09:38:21 (1782481101) [ 731.973583] Key type .llcrypt unregistered [ 731.976540] Key type ._llcrypt unregistered [ 738.397541] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 738.524655] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 738.527662] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 738.540529] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 741.206297] Key type ._llcrypt registered [ 741.209703] Key type .llcrypt registered [ 741.247331] LNet: 33933:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 741.247331] mask usage: [+|-] ... [ 741.382822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 741.393936] alg: No test for adler32 (adler32-zlib) [ 742.295976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.347439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 742.390255] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 743.253685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 743.274715] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 745.475275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 745.499156] LNet: Removed LNI 10.1.2.3@tcp [ 745.642859] LNet: 34376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.650873] LNetError: 34376:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.667870] LNet: Removed LNI 192.168.204.52@tcp [ 746.306675] Key type .llcrypt unregistered [ 746.313758] Key type ._llcrypt unregistered [ 758.838941] Key type ._llcrypt registered [ 758.841638] Key type .llcrypt registered [ 760.549460] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 09:38:50 (1782481130) [ 761.112455] Key type .llcrypt unregistered [ 761.115368] Key type ._llcrypt unregistered [ 767.700145] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 767.813929] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 767.818446] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 767.830261] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 770.684462] Key type ._llcrypt registered [ 770.686476] Key type .llcrypt registered [ 770.762684] LNet: 35414:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 770.762684] mask usage: [+|-] ... [ 770.966818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 770.991436] alg: No test for adler32 (adler32-zlib) [ 771.936761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.986404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 772.006471] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 772.679722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 772.710669] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 775.522223] LNet: 35872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.535091] LNetError: 35872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 775.541821] LNet: Removed LNI 10.1.2.3@tcp [ 775.560421] LNet: Removed LNI 192.168.204.52@tcp [ 776.084337] Key type .llcrypt unregistered [ 776.087734] Key type ._llcrypt unregistered [ 787.511431] Key type ._llcrypt registered [ 787.515564] Key type .llcrypt registered [ 788.774921] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 09:39:19 (1782481159) [ 789.447402] Key type .llcrypt unregistered [ 789.453067] Key type ._llcrypt unregistered [ 796.006588] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 796.263106] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 796.271501] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 796.282880] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 798.026782] Key type ._llcrypt registered [ 798.035649] Key type .llcrypt registered [ 798.109402] LNet: 36901:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 798.109402] mask usage: [+|-] ... [ 798.383620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 798.416444] alg: No test for adler32 (adler32-zlib) [ 799.424437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.527837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 799.556514] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 801.975645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 802.007824] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 804.645176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 804.687659] LNet: Removed LNI 10.1.2.3@tcp [ 804.714954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 804.792524] LNet: Removed LNI 10.1.2.31@tcp [ 804.822875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 804.858581] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 805.765904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 805.781411] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 806.785857] LNet: 37784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.792275] LNetError: 37784:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 806.795935] LNet: Removed LNI 10.1.2.31@tcp [ 806.804202] LNet: Removed LNI 10.1.2.3@tcp [ 807.395481] Key type .llcrypt unregistered [ 807.398470] Key type ._llcrypt unregistered [ 821.044406] Key type ._llcrypt registered [ 821.046681] Key type .llcrypt registered [ 822.758950] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:39:52 (1782481192) [ 823.068270] LNet: 38577:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 823.068270] mask usage: [+|-] ... [ 823.265518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 823.305164] alg: No test for adler32 (adler32-zlib) [ 824.194604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.262702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 824.301982] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 825.026837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 825.042910] LNet: 38738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.047987] LNetError: 38738:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.059933] LNet: Removed LNI 192.168.204.52@tcp [ 825.094562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.150962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.52@tcp [ 825.180637] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 826.217601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.52@tcp [ 826.259871] LNet: Removed LNI 192.168.204.52@tcp [ 826.360886] LNet: 38873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.051537] Key type .llcrypt unregistered [ 827.058233] Key type ._llcrypt unregistered [ 838.863398] Key type ._llcrypt registered [ 838.867951] Key type .llcrypt registered [ 840.362809] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:40:10 (1782481210) [ 842.454736] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg452-server [ 844.337023] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:40:15 (1782481215) [ 844.948319] Key type .llcrypt unregistered [ 844.952517] Key type ._llcrypt unregistered [ 851.703857] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 851.859950] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 851.874109] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 851.893198] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 854.517450] Key type ._llcrypt registered [ 854.520068] Key type .llcrypt registered [ 854.555981] LNet: 40250:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 854.555981] mask usage: [+|-] ... [ 854.717189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 854.749845] alg: No test for adler32 (adler32-zlib) [ 855.671500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.778377] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 856.841088] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 859.123212] LNet: 40710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.131385] LNetError: 40710:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.143560] LNet: Removed LNI 192.168.204.52@tcp1 [ 859.155060] LNet: Removed LNI 10.1.2.3@tcp2 [ 859.791920] Key type .llcrypt unregistered [ 859.797155] Key type ._llcrypt unregistered [ 873.636412] Key type ._llcrypt registered [ 873.639626] Key type .llcrypt registered [ 875.137726] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:40:45 (1782481245) [ 875.460967] LNet: 41514:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 875.460967] mask usage: [+|-] ... [ 875.651250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 875.685881] alg: No test for adler32 (adler32-zlib) [ 876.639234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.698713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 876.727740] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 877.522220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.547789] LNet: 41679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.558729] LNetError: 41679:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.573568] LNet: Removed LNI 192.168.204.52@tcp [ 877.607935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.691750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 877.724808] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 884.729266] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:40:55 (1782481255) [ 885.172161] LNet: 42270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.194694] LNetError: 42270:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.225901] LNet: Removed LNI 192.168.204.52@tcp [ 885.953374] Key type .llcrypt unregistered [ 885.955703] Key type ._llcrypt unregistered [ 893.083401] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 893.202650] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 893.211891] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 893.221848] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 895.895740] Key type ._llcrypt registered [ 895.898740] Key type .llcrypt registered [ 895.943535] LNet: 42548:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 895.943535] mask usage: [+|-] ... [ 896.137947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 896.157287] alg: No test for adler32 (adler32-zlib) [ 897.108190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.178843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 897.235233] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 898.418162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 898.444066] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 900.835747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.855860] LNet: 43014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.862336] LNetError: 43014:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 900.866708] LNet: Removed LNI 10.1.2.3@tcp [ 900.873491] LNet: Removed LNI 192.168.204.52@tcp [ 900.896314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.953305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 900.973870] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 900.977978] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 901.585032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.609698] LNet: 43042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.619406] LNetError: 43042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 901.625841] LNetError: 43042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 901.638617] LNet: Removed LNI 10.1.2.3@tcp [ 901.649267] LNet: Removed LNI 192.168.204.52@tcp [ 901.664057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.713585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 901.730947] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 901.734566] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 902.294700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 902.333755] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 903.314654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 903.338291] LNet: 43204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.344891] LNetError: 43204:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 903.350329] LNetError: 43204:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 903.355463] LNet: Removed LNI 10.1.2.3@tcp [ 903.358185] LNet: Removed LNI 192.168.204.52@tcp [ 903.368694] LNet: Removed LNI 192.168.204.52@tcp2 [ 903.385888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.464372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 903.485799] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 903.491125] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 903.496924] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 911.649393] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 09:41:22 (1782481282) [ 911.858550] LNet: 43796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.879910] LNetError: 43796:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 911.888543] LNetError: 43796:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 911.899611] LNet: Removed LNI 10.1.2.3@tcp [ 911.911605] LNet: Removed LNI 192.168.204.52@tcp [ 911.937449] LNet: Removed LNI 192.168.204.52@tcp2 [ 912.799392] Key type .llcrypt unregistered [ 912.803877] Key type ._llcrypt unregistered [ 919.925612] Key type ._llcrypt registered [ 919.928405] Key type .llcrypt registered [ 919.991910] LNet: 44029:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 919.991910] mask usage: [+|-] ... [ 920.213282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 920.261848] alg: No test for adler32 (adler32-zlib) [ 921.256804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.343643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 921.380670] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 922.509398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.535263] LNet: 44198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.550627] LNetError: 44198:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.582907] LNet: Removed LNI 192.168.204.52@tcp [ 922.614340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.705485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 922.748895] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 923.416609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 923.450567] LNet: 44228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.463456] LNetError: 44228:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.495819] LNet: Removed LNI 192.168.204.52@tcp [ 923.536975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.657174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 923.734822] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 924.418366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.449766] LNet: 44259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.464750] LNet: Removed LNI 192.168.204.52@tcp [ 924.483915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.572575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 924.635080] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 925.522376] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 925.739836] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 925.754202] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 925.763212] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 928.401707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 928.439213] LNet: 44332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.447619] LNetError: 44332:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.460887] LNetError: 44332:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 928.499198] LNet: Removed LNI 192.168.204.52@tcp [ 928.524250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.642160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 928.713130] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 938.344036] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 09:41:48 (1782481308) [ 938.480468] LNet: 44921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.488257] LNetError: 44921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.513152] LNet: Removed LNI 192.168.204.52@tcp [ 939.520970] Key type .llcrypt unregistered [ 939.524064] Key type ._llcrypt unregistered [ 948.175351] Key type ._llcrypt registered [ 948.180586] Key type .llcrypt registered [ 948.276976] LNet: 45156:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 948.276976] mask usage: [+|-] ... [ 948.551324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 948.571788] alg: No test for adler32 (adler32-zlib) [ 949.564944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.674979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 949.744529] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 950.790340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.823798] LNet: 45329:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.833945] LNetError: 45329:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.851490] LNet: Removed LNI 192.168.204.52@tcp [ 950.890698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.955699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 951.008330] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 959.224366] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 09:42:09 (1782481329) [ 959.355385] LNet: 45914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.366270] LNetError: 45914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.405284] LNet: Removed LNI 192.168.204.52@tcp [ 960.578888] Key type .llcrypt unregistered [ 960.583042] Key type ._llcrypt unregistered [ 968.791331] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 968.925070] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 968.930947] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 968.938294] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 971.439244] Key type ._llcrypt registered [ 971.441791] Key type .llcrypt registered [ 971.471799] LNet: 46199:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 971.471799] mask usage: [+|-] ... [ 971.672747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 971.699799] alg: No test for adler32 (adler32-zlib) [ 972.653841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 972.749554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 972.794759] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 973.681834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 973.702255] LNet: 46373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.708175] LNetError: 46373:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 973.721794] LNet: Removed LNI 192.168.204.52@tcp [ 973.737839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.809603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 973.831603] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 981.546766] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 09:42:31 (1782481351) [ 981.638324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.673787] LNet: 46967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.680640] LNetError: 46967:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.708791] LNet: Removed LNI 192.168.204.52@tcp [ 981.767520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.909425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 988.254621] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 09:42:38 (1782481358) [ 988.423666] LNet: 47551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.119343] Key type .llcrypt unregistered [ 989.121258] Key type ._llcrypt unregistered [ 996.556309] Key type ._llcrypt registered [ 996.558954] Key type .llcrypt registered [ 996.592314] LNet: 47780:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 996.592314] mask usage: [+|-] ... [ 996.774735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 996.799704] alg: No test for adler32 (adler32-zlib) [ 997.759038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.874859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 997.927257] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 998.787245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.804627] LNet: 47952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.809623] LNetError: 47952:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.828690] LNet: Removed LNI 192.168.204.52@tcp [ 998.844970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.902954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 998.924138] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1005.923679] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 09:42:56 (1782481376) [ 1006.091726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1006.125522] LNet: 48545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.130301] LNetError: 48545:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.144891] LNet: Removed LNI 192.168.204.52@tcp [ 1006.167808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.223814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1012.511673] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 09:43:03 (1782481383) [ 1014.043686] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1015.639318] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 09:43:06 (1782481386) [ 1017.292118] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1019.183730] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 09:43:09 (1782481389) [ 1020.746950] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1022.430552] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 09:43:13 (1782481393) [ 1023.999470] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1026.179851] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 09:43:16 (1782481396) [ 1027.958775] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1029.697340] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 09:43:20 (1782481400) [ 1029.788739] LNet: 50589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.536275] Key type .llcrypt unregistered [ 1030.540274] Key type ._llcrypt unregistered [ 1036.908326] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1036.983587] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1036.987490] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1036.990565] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1039.813495] Key type ._llcrypt registered [ 1039.816763] Key type .llcrypt registered [ 1039.867477] LNet: 50881:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1039.867477] mask usage: [+|-] ... [ 1040.120656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1040.166040] alg: No test for adler32 (adler32-zlib) [ 1041.124905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.216811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1041.246441] LNet: Added LNI 192.168.204.52@tcp [8/512/0/180] [ 1041.254940] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1041.258419] LNet: 50919:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1041.386806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1041.410915] LNet: 50938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.420293] LNetError: 50938:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1041.432009] LNet: Removed LNI 10.1.2.3@tcp [ 1041.433640] LNetError: 50938:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.453577] LNet: Removed LNI 192.168.204.52@tcp [ 1041.472665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.563743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1041.611352] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1041.752755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1041.813951] LNet: 50962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.822395] LNetError: 50962:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.858761] LNet: Removed LNI 192.168.204.52@tcp [ 1041.895875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.975271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1041.993926] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1042.006387] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1042.009800] LNet: 50970:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1042.136800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.159513] LNet: 50988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.165630] LNetError: 50988:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1042.174862] LNet: Removed LNI 10.1.2.3@tcp [ 1042.179234] LNetError: 50988:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.192276] LNet: Removed LNI 192.168.204.52@tcp [ 1042.213825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.284658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1042.309133] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1042.322124] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1042.329602] LNet: 50996:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1042.579433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.596358] LNet: 51022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.601485] LNetError: 51022:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1042.608509] LNet: Removed LNI 10.1.2.3@tcp [ 1042.610500] LNetError: 51022:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.624986] LNet: Removed LNI 192.168.204.52@tcp [ 1042.654837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.717874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1042.740066] LNet: Added LNI 192.168.204.52@tcp [8/256/0/182] [ 1042.746034] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1042.842686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.867695] LNet: 51050:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.876919] LNetError: 51050:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1042.882758] LNet: Removed LNI 10.1.2.3@tcp [ 1042.885636] LNetError: 51050:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.903296] LNet: Removed LNI 192.168.204.52@tcp [ 1042.924839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.997679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1043.039506] LNet: Added LNI 192.168.204.52@tcp [8/256/0/182] [ 1043.046324] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1043.052378] LNet: 51058:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1043.150477] LNet: Removed LNI 10.1.2.3@tcp [ 1043.165601] LNet: Removed LNI 192.168.204.52@tcp [ 1043.759397] Key type .llcrypt unregistered [ 1043.762983] Key type ._llcrypt unregistered [ 1056.209197] Key type ._llcrypt registered [ 1056.212671] Key type .llcrypt registered [ 1057.434577] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:43:48 (1782481428) [ 1057.773111] LNet: 51876:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1057.773111] mask usage: [+|-] ... [ 1057.968073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1057.988660] alg: No test for adler32 (adler32-zlib) [ 1058.939930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.995804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1059.016919] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1059.743754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.52 [ 1059.760461] LNet: 52038:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1059.767512] LNet: Removed LNI 192.168.204.52@tcp [ 1066.405110] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:43:57 (1782481437) [ 1066.497464] LNet: 52599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.504804] LNetError: 52599:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.528078] LNet: Removed LNI 192.168.204.52@tcp [ 1067.222689] Key type .llcrypt unregistered [ 1067.225823] Key type ._llcrypt unregistered [ 1073.269309] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1073.539173] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1076.280706] Key type ._llcrypt registered [ 1076.282879] Key type .llcrypt registered [ 1076.316402] LNet: 52890:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1076.316402] mask usage: [+|-] ... [ 1076.475328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1076.499919] alg: No test for adler32 (adler32-zlib) [ 1077.834684] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1083.694836] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:44:14 (1782481454) [ 1083.842875] LNet: 53520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.853799] LNetError: 53520:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1083.883654] LNet: Removed LNI 10.1.2.3@tcp [ 1084.529489] Key type .llcrypt unregistered [ 1084.531842] Key type ._llcrypt unregistered [ 1090.356709] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1090.643685] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1090.648988] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1090.656640] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1093.442297] Key type ._llcrypt registered [ 1093.444511] Key type .llcrypt registered [ 1093.467582] LNet: 53795:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1093.467582] mask usage: [+|-] ... [ 1093.598404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1093.613892] alg: No test for adler32 (adler32-zlib) [ 1094.554113] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1101.087301] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:44:31 (1782481471) [ 1102.699873] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1104.385756] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:44:35 (1782481475) [ 1104.541342] LNet: 54702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1104.552576] LNetError: 54702:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1104.583625] LNet: Removed LNI 10.1.2.3@tcp [ 1105.090718] Key type .llcrypt unregistered [ 1105.092625] Key type ._llcrypt unregistered [ 1111.111450] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1111.352334] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1111.356442] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1111.360837] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1113.986364] Key type ._llcrypt registered [ 1113.988903] Key type .llcrypt registered [ 1114.036421] LNet: 54981:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1114.036421] mask usage: [+|-] ... [ 1114.203468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1114.225674] alg: No test for adler32 (adler32-zlib) [ 1115.180989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1115.269810] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1123.641865] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:44:54 (1782481494) [ 1124.055266] LNet: 55936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1124.060841] LNetError: 55936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1124.080077] LNet: Removed LNI 10.1.2.3@tcp [ 1124.581725] Key type .llcrypt unregistered [ 1124.586787] Key type ._llcrypt unregistered [ 1129.468564] Key type ._llcrypt registered [ 1129.472798] Key type .llcrypt registered [ 1129.511465] LNet: 56159:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1129.511465] mask usage: [+|-] ... [ 1129.644683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1129.664566] alg: No test for adler32 (adler32-zlib) [ 1130.551515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1130.584585] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1137.010464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1137.738957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_interrupt -r 1 [ 1137.796469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_interrupt -r 1 [ 1137.842821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1137.868875] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1138.104621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1138.145366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1138.203199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1138.253759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1138.294949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1138.347874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1138.390499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1138.425569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1138.446856] LNet: 56374:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1138.483209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1138.514546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1138.551943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1138.589449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1138.627194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1138.660227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1138.696822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1139.225644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_dropped -r 1 [ 1139.277221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_dropped -r 1 [ 1139.348422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1139.380035] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1139.631193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1139.662979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1139.691105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1139.727644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1139.753185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1139.790182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1139.808049] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1139.820201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1139.855435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1139.875819] LNet: 56483:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1139.892514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1139.916993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1139.950297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1139.983060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.016735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1140.044717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1140.080844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1140.561760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_aborted -r 1 [ 1140.609583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_aborted -r 1 [ 1140.652253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1140.671724] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1140.877278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.903721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1140.933721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1140.971102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1141.005707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1141.049389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1141.093193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1141.141544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1141.168684] LNet: 56592:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1141.196242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1141.230409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1141.274856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1141.309033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1141.344347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1141.384451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1141.412683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1142.000671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_no_route -r 1 [ 1142.046791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_no_route -r 1 [ 1142.086241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1142.371199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1142.395991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1142.441033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1142.488386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1142.517685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1142.562840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1142.604920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1142.646259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1142.678338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1142.716543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1142.783512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1142.836383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1142.869607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1142.906290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1142.934749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1143.424736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_timeout -r 1 [ 1143.465512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_timeout -r 1 [ 1143.503204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1143.527059] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1143.532707] LNet: Skipped 2 previous similar messages [ 1143.723940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1143.762628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1143.815129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1143.851852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1143.883602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1143.911384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1143.936435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1143.958112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1143.974503] LNet: 56810:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1143.979569] LNet: 56810:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1143.992937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1144.013359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1144.046311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1144.072802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1144.100995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1144.131877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1144.162747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1144.611576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_error -r 1 [ 1144.656217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_error -r 1 [ 1144.680796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1144.875125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1144.910417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1144.955803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1144.998474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1145.030273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1145.079584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1145.115509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1145.143123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1145.188800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1145.229641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1145.259100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1145.290900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1145.320913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1145.351666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1145.372954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1145.562980] LNet: 56961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.569694] LNetError: 56961:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1146.613355] LNet: Removed LNI 192.168.204.52@tcp [ 1147.061376] Key type .llcrypt unregistered [ 1147.065097] Key type ._llcrypt unregistered [ 1164.291981] Key type ._llcrypt registered [ 1164.293586] Key type .llcrypt registered [ 1165.378908] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:45:36 (1782481536) [ 1166.048548] Key type .llcrypt unregistered [ 1166.050910] Key type ._llcrypt unregistered [ 1171.480869] Key type ._llcrypt registered [ 1171.483775] Key type .llcrypt registered [ 1171.521531] LNet: 57998:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1171.521531] mask usage: [+|-] ... [ 1171.645740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1171.658802] alg: No test for adler32 (adler32-zlib) [ 1172.579779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1172.603219] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1179.474564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1187.236588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1187.261318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1187.291086] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1188.423307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_interrupt -r 1 [ 1188.473083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_interrupt -r 1 [ 1188.507819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_interrupt -r 1 [ 1188.562294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_interrupt -r 1 [ 1188.600970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_interrupt -r 1 [ 1188.638015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_interrupt -r 1 [ 1188.677216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_interrupt -r 1 [ 1188.714997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_interrupt -r 1 [ 1188.749748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1188.774724] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1188.786370] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1188.984859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.021108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1189.056921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1189.102939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1189.145394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1189.188975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1189.221519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1189.264444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1189.286051] LNet: 58426:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1189.302517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1189.342678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1189.381895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1189.418831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1189.449949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.494482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.530870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1189.958888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_dropped -r 1 [ 1189.993811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_dropped -r 1 [ 1190.038578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_dropped -r 1 [ 1190.082574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_dropped -r 1 [ 1190.137095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_dropped -r 1 [ 1190.184620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_dropped -r 1 [ 1190.237501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_dropped -r 1 [ 1190.282583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_dropped -r 1 [ 1190.324535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1190.338640] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1190.343122] LNet: Skipped 2 previous similar messages [ 1190.534747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1190.566507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1190.593559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1190.618748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1190.653756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1190.690849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1190.730676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1190.757803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1190.777465] LNet: 58547:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1190.794535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1190.829701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1190.858992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1190.903831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.946613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.984917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1191.016963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1191.576492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_aborted -r 1 [ 1191.621776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_aborted -r 1 [ 1191.665652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_aborted -r 1 [ 1191.708087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_aborted -r 1 [ 1191.750268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_aborted -r 1 [ 1191.780772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_aborted -r 1 [ 1191.812750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_aborted -r 1 [ 1191.836500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_aborted -r 1 [ 1191.876348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1191.895832] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1191.902265] LNet: Skipped 2 previous similar messages [ 1192.105488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1192.133253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1192.168582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1192.206149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1192.237478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1192.266288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1192.320163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1192.357468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1192.392550] LNet: 58669:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1192.420313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1192.455504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1192.483526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1192.516907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1192.562696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.599399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1192.628302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1193.072273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_no_route -r 1 [ 1193.127684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_no_route -r 1 [ 1193.161666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_no_route -r 1 [ 1193.204295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_no_route -r 1 [ 1193.236466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_no_route -r 1 [ 1193.269903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_no_route -r 1 [ 1193.302823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_no_route -r 1 [ 1193.332782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_no_route -r 1 [ 1193.392238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1193.616215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1193.649814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1193.682829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1193.710851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1193.742369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1193.777364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1193.816481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1193.844964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1193.872168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1193.894824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1193.919584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1193.969181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1194.000146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1194.040934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1194.083886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1194.508205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_timeout -r 1 [ 1194.541849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_timeout -r 1 [ 1194.575495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_timeout -r 1 [ 1194.614378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_timeout -r 1 [ 1194.668169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_timeout -r 1 [ 1194.734172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_timeout -r 1 [ 1194.778853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_timeout -r 1 [ 1194.830451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_timeout -r 1 [ 1194.879901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1194.904731] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1194.913099] LNet: Skipped 5 previous similar messages [ 1195.124817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1195.166154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1195.203196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1195.253594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1195.286339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1195.310896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1195.348665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1195.378481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1195.390846] LNet: 58911:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1195.395781] LNet: 58911:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1195.407937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1195.437177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1195.461641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1195.492563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1195.519313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1195.549910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1195.600461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1196.059442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_error -r 1 [ 1196.093848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_error -r 1 [ 1196.132590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_error -r 1 [ 1196.160451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_error -r 1 [ 1196.190732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_error -r 1 [ 1196.229849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_error -r 1 [ 1196.272779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_error -r 1 [ 1196.306742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_error -r 1 [ 1196.349981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1196.599584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1196.632868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1196.653468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1196.688413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1196.713390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1196.738079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1196.767743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1196.793861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1196.823510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1196.855840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1196.878982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1196.913129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1196.947081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1197.001399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1197.025696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1201.882628] LNet: 59108:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1201.888835] LNetError: 59108:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1202.912171] LNet: Removed LNI 192.168.204.52@tcp [ 1202.917222] LNetError: 59108:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1202.958274] LNet: Removed LNI 192.168.204.52@tcp1 [ 1203.425670] Key type .llcrypt unregistered [ 1203.427640] Key type ._llcrypt unregistered [ 1219.430393] Key type ._llcrypt registered [ 1219.433717] Key type .llcrypt registered [ 1220.533366] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:46:31 (1782481591) [ 1221.326815] Key type .llcrypt unregistered [ 1221.330842] Key type ._llcrypt unregistered [ 1227.571389] Key type ._llcrypt registered [ 1227.575056] Key type .llcrypt registered [ 1227.626060] LNet: 60143:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1227.626060] mask usage: [+|-] ... [ 1227.795631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1227.816714] alg: No test for adler32 (adler32-zlib) [ 1228.736468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1228.777680] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1235.553684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1236.099469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_dropped -r 1 [ 1236.136253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_dropped -r 1 [ 1236.176524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1236.205164] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1236.394204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.437310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1236.463353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.501224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.545711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.591644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.616646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.645567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.673328] LNet: 60357:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.704145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.738958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.779132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.819813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.865418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.904384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1236.943923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1237.471532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_error -r 1 [ 1237.502572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_error -r 1 [ 1237.535793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1237.554788] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1237.763786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1237.787831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1237.809883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1237.832691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1237.869963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1237.896797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1237.930163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1237.960124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1237.988192] LNet: 60466:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1238.005461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1238.031850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1238.082945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1238.124177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.188867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.237928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.286214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1238.849544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_timeout -r 1 [ 1238.895901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_timeout -r 1 [ 1238.920774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1238.941968] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1239.219682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1239.259490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1239.286618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1239.314448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1239.352233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1239.385707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1239.425337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1239.454897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1239.475194] LNet: 60576:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1239.494128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1239.520882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1239.572374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1239.610820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1239.635927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1239.658519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1239.681146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1239.886267] LNet: 60617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1239.891299] LNetError: 60617:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1239.901953] LNet: Removed LNI 192.168.204.52@tcp [ 1240.261297] Key type .llcrypt unregistered [ 1240.264311] Key type ._llcrypt unregistered [ 1253.605741] Key type ._llcrypt registered [ 1253.607658] Key type .llcrypt registered [ 1254.628598] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:47:05 (1782481625) [ 1255.242622] Key type .llcrypt unregistered [ 1255.245345] Key type ._llcrypt unregistered [ 1260.440320] Key type ._llcrypt registered [ 1260.442797] Key type .llcrypt registered [ 1260.471336] LNet: 61653:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1260.471336] mask usage: [+|-] ... [ 1260.627860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1260.638642] alg: No test for adler32 (adler32-zlib) [ 1261.535640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1261.555665] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1267.853907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1275.553431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1275.626153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1275.645909] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1276.641748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_dropped -r 1 [ 1276.669923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e remote_dropped -r 1 [ 1276.704552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_dropped -r 1 [ 1276.735343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e remote_dropped -r 1 [ 1276.776266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e remote_dropped -r 1 [ 1276.813294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e remote_dropped -r 1 [ 1276.861578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e remote_dropped -r 1 [ 1276.907873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e remote_dropped -r 1 [ 1276.947952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1276.964726] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1276.972376] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.152@tcp [ 1277.126466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1277.162677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1277.196752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1277.217709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1277.247390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1277.269643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1277.301852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1277.328829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1277.347480] LNet: 62081:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1277.374947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1277.437079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1277.470889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1277.495242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1277.523920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1277.549390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1277.581385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1277.610574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.003919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_error -r 1 [ 1278.047818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e remote_error -r 1 [ 1278.082858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_error -r 1 [ 1278.110480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e remote_error -r 1 [ 1278.149544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e remote_error -r 1 [ 1278.190766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e remote_error -r 1 [ 1278.224426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e remote_error -r 1 [ 1278.259898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e remote_error -r 1 [ 1278.293637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1278.311869] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1278.315680] LNet: Skipped 2 previous similar messages [ 1278.490616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1278.515893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1278.540917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1278.571650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1278.598852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1278.631900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.671794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1278.701131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.730918] LNet: 62204:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.749893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.781482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.813996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.841690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.874834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.914167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1278.960944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1279.002062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1279.387726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_timeout -r 1 [ 1279.414794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e remote_timeout -r 1 [ 1279.444237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_timeout -r 1 [ 1279.475182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e remote_timeout -r 1 [ 1279.501832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e remote_timeout -r 1 [ 1279.527062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e remote_timeout -r 1 [ 1279.553222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e remote_timeout -r 1 [ 1279.585590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e remote_timeout -r 1 [ 1279.622524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1279.649689] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1279.835083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1279.874272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1279.901495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.928680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1279.962406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.009618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.059357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.099643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.114402] LNet: 62328:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.134505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.159843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.198816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.225355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.255921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.278850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1280.303519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1280.327557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1283.375820] LNet: 62405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1283.384546] LNetError: 62405:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1283.396060] LNet: Removed LNI 192.168.204.52@tcp [ 1283.405803] LNet: Removed LNI 192.168.204.52@tcp1 [ 1283.866440] Key type .llcrypt unregistered [ 1283.870833] Key type ._llcrypt unregistered [ 1297.955356] Key type ._llcrypt registered [ 1297.957834] Key type .llcrypt registered [ 1299.028448] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:47:49 (1782481669) [ 1300.582516] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1302.224471] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:47:52 (1782481672) [ 1303.010624] Key type .llcrypt unregistered [ 1303.013466] Key type ._llcrypt unregistered [ 1307.671308] Key type ._llcrypt registered [ 1307.673506] Key type .llcrypt registered [ 1307.708601] LNet: 63733:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1307.708601] mask usage: [+|-] ... [ 1307.832741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1307.848680] alg: No test for adler32 (adler32-zlib) [ 1308.748722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1308.772141] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1314.223652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1314.722813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e network_timeout -r 1 [ 1314.748342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e network_timeout -r 1 [ 1314.779199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1314.796961] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1314.911125] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1314.967818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1314.999943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1315.035918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1315.066829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1315.100179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1315.129902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1315.153182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1315.176718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1315.192723] LNet: 63947:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1315.210095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1315.231736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1315.261256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1315.280822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1315.305149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1315.344438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1315.514861] LNet: 63986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1315.526491] LNetError: 63986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1315.548806] LNet: Removed LNI 192.168.204.52@tcp [ 1315.857042] Key type .llcrypt unregistered [ 1315.861052] Key type ._llcrypt unregistered [ 1328.144863] Key type ._llcrypt registered [ 1328.147832] Key type .llcrypt registered [ 1328.178453] LNet: 64435:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1328.178453] mask usage: [+|-] ... [ 1328.285739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1328.302300] alg: No test for adler32 (adler32-zlib) [ 1329.236792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1329.257337] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1335.723451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1342.943779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1342.997421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1343.023388] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1344.447661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e network_timeout -r 1 [ 1344.509587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e network_timeout -r 1 [ 1344.550144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e network_timeout -r 1 [ 1344.580264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e network_timeout -r 1 [ 1344.611962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e network_timeout -r 1 [ 1344.645238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e network_timeout -r 1 [ 1344.683807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e network_timeout -r 1 [ 1344.722827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e network_timeout -r 1 [ 1344.758917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1344.796947] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1345.025531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1345.083283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1345.131193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1345.180159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1345.211559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1345.242371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1345.270697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1345.312840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1345.331846] LNet: 64863:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1345.354225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1345.385415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1345.412875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1345.438961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1345.470198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1345.495437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1349.307841] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1349.588653] LNet: 64937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.595476] LNetError: 64937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.601433] LNet: Removed LNI 192.168.204.52@tcp [ 1349.619712] LNet: Removed LNI 192.168.204.52@tcp1 [ 1350.024433] Key type .llcrypt unregistered [ 1350.026603] Key type ._llcrypt unregistered [ 1363.240361] Key type ._llcrypt registered [ 1363.244584] Key type .llcrypt registered [ 1364.119598] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:48:55 (1782481735) [ 1364.350321] LNet: 65737:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1364.350321] mask usage: [+|-] ... [ 1364.474858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1364.491563] alg: No test for adler32 (adler32-zlib) [ 1365.377303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1365.430089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1365.454900] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1366.037162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1366.055298] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1366.747631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1366.817316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1366.860736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1366.908442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1366.940291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1366.968958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1366.992654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.52@tcp [ 1367.023383] LNetError: 66036:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.52@tcp to recovery [ 1367.455330] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1367.456104] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1369.504083] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1370.416462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1373.599728] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1373.806206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1373.833449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1373.890766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1373.909520] LNet: 66143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.921469] LNetError: 66143:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1373.929458] LNet: Removed LNI 192.168.204.52@tcp [ 1373.944022] LNet: Removed LNI 192.168.204.52@tcp1 [ 1373.960713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.000991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1374.019814] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1374.683391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1374.703272] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1375.338970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1375.362884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1375.405786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1375.443861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1375.472823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1375.505102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1375.544390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1375.575120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.52@tcp [ 1375.601608] LNetError: 66416:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.52@tcp to recovery [ 1376.032359] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1379.023831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1382.175818] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1382.185344] LNet: Skipped 1 previous similar message [ 1386.714714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1386.750641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1386.777134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1392.027219] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 09:49:22 (1782481762) [ 1392.148871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1392.173126] LNet: 67126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1392.178860] LNetError: 67126:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.190785] LNetError: 67126:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1392.195799] LNet: Removed LNI 192.168.204.52@tcp [ 1392.210891] LNet: Removed LNI 192.168.204.52@tcp1 [ 1392.224796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1392.294817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1392.317918] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1392.849496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1392.867560] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1393.542684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1393.602826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1393.627345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1393.654762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1393.680807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1393.705649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1393.729828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.52@tcp --health 0 [ 1393.821893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1394.335772] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.52@tcp [ 1394.335835] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1394.344646] LNet: Skipped 1 previous similar message [ 1409.086972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1409.213813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1409.243170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1409.278685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.52@tcp --health 500 [ 1409.363867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1416.958381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1417.024444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1417.054961] LNet: 67632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1417.062560] LNetError: 67632:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1417.073617] LNetError: 67632:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1417.081804] LNet: Removed LNI 192.168.204.52@tcp [ 1417.092537] LNet: Removed LNI 192.168.204.52@tcp1 [ 1417.120496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1417.211451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1417.229806] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1418.078689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1418.104493] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1418.825599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1418.856604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1418.908223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1418.935970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1418.973963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1419.023529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1419.068799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1419.094311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.52@tcp --health 0 [ 1420.257450] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1420.267416] LNet: Skipped 3 previous similar messages [ 1421.468524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1431.444817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1431.479435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1431.509217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1436.780953] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 09:50:07 (1782481807) [ 1437.215703] LNet: 68688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1437.226931] LNetError: 68688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1437.232682] LNetError: 68688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1437.241216] LNet: Removed LNI 192.168.204.52@tcp [ 1437.254398] LNet: Removed LNI 192.168.204.52@tcp1 [ 1437.729505] Key type .llcrypt unregistered [ 1437.736339] Key type ._llcrypt unregistered [ 1442.124808] Key type ._llcrypt registered [ 1442.128732] Key type .llcrypt registered [ 1442.170341] LNet: 68910:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1442.170341] mask usage: [+|-] ... [ 1442.303631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1442.318450] alg: No test for adler32 (adler32-zlib) [ 1443.220178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1443.242494] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1450.124222] Lustre: DEBUG MARKER: Initial discovery [ 1450.198517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.152@tcp [ 1451.846556] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1451.965544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1452.008771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.152@tcp [ 1452.035518] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1452.062948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1452.104978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.152@tcp -r 1 -m GET -l 3 [ 1452.143858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.152@tcp -r 1 -m GET -e local_error [ 1452.173926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.152@tcp -r 1 -m PUT -l 6 [ 1452.447112] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1453.219783] Lustre: DEBUG MARKER: Force oleg452-client.virtnet to discover 192.168.204.152@tcp (in background) [ 1453.401824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.152@tcp [ 1454.361483] Lustre: DEBUG MARKER: Force oleg452-server to discover 192.168.204.52@tcp [ 1456.276558] Lustre: DEBUG MARKER: Wait for 69450 [ 1456.479315] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1457.499676] Lustre: DEBUG MARKER: Finished wait on 69450 [ 1457.678745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1457.703649] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1457.705201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1457.930705] LNet: 69901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.940568] LNetError: 69901:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1457.959858] LNet: Removed LNI 192.168.204.52@tcp [ 1458.231382] Key type .llcrypt unregistered [ 1458.233185] Key type ._llcrypt unregistered [ 1469.617167] Key type ._llcrypt registered [ 1469.620037] Key type .llcrypt registered [ 1470.866926] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 09:50:41 (1782481841) [ 1471.351557] Key type .llcrypt unregistered [ 1471.356521] Key type ._llcrypt unregistered [ 1476.382496] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1476.448659] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1476.453780] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1476.459206] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1479.063609] Key type ._llcrypt registered [ 1479.066633] Key type .llcrypt registered [ 1479.118421] LNet: 70944:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1479.118421] mask usage: [+|-] ... [ 1479.314437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1479.336633] alg: No test for adler32 (adler32-zlib) [ 1480.216346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.262460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1480.278904] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1480.922350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1480.947905] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1486.866554] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 09:50:58 (1782481858) [ 1486.961163] LNet: 71971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1486.970564] LNetError: 71971:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1486.978383] LNet: Removed LNI 10.1.2.3@tcp [ 1486.991487] LNet: Removed LNI 192.168.204.52@tcp [ 1487.402682] Key type .llcrypt unregistered [ 1487.406708] Key type ._llcrypt unregistered [ 1491.789549] Key type ._llcrypt registered [ 1491.792748] Key type .llcrypt registered [ 1491.822945] LNet: 72226:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1491.822945] mask usage: [+|-] ... [ 1491.951577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1491.963854] alg: No test for adler32 (adler32-zlib) [ 1492.851907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.897589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1492.920687] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1493.484313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1493.504493] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1494.385608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.52@tcp [ 1495.455840] LNetError: Unexpected error -22 connecting to 192.168.204.52@tcp at host 192.168.204.52:988 [ 1499.613659] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 09:51:10 (1782481870) [ 1499.949601] LNet: 73108:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1499.955413] LNetError: 73108:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1499.968495] LNet: Removed LNI 10.1.2.3@tcp [ 1499.979923] LNet: Removed LNI 192.168.204.52@tcp [ 1500.533162] Key type .llcrypt unregistered [ 1500.536848] Key type ._llcrypt unregistered [ 1505.915164] Key type ._llcrypt registered [ 1505.916728] Key type .llcrypt registered [ 1505.943912] LNet: 73338:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1505.943912] mask usage: [+|-] ... [ 1506.085970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1506.097264] alg: No test for adler32 (adler32-zlib) [ 1506.957791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1507.004292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1507.033316] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1507.492489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1507.522836] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 1508.163839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.52@tcp1 --nid 192.168.204.52@tcp2 [ 1511.269161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp1 192.168.204.52@tcp2 [ 1511.294680] LNetError: 73941:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.52@tcp1 to 192.168.204.52@tcp2 - Route table may be misconfigured [ 1511.313808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp2 192.168.204.52@tcp1 [ 1516.000785] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 09:51:26 (1782481886) [ 1516.159669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.192737] LNet: 74515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1516.203293] LNetError: 74515:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1516.209860] LNet: Removed LNI 192.168.204.52@tcp1 [ 1517.221635] LNet: Removed LNI 192.168.204.52@tcp2 [ 1517.243919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.289758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1517.321652] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1517.936960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1517.952614] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1518.651662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1518.689414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.52@tcp [ 1518.736769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp1 192.168.204.52@tcp1 [ 1518.795644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.52@tcp -d 192.168.204.52@tcp -e network_timeout [ 1518.825385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -e network_timeout [ 1518.857343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -e network_timeout [ 1518.887524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -e network_timeout [ 1518.937643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1518.958136] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1518.980683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1519.062713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1519.167821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1519.329401] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1524.265532] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:51:35 (1782481895) [ 1524.348704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1524.378630] LNet: 75369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1524.390472] LNetError: 75369:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1524.397825] LNetError: 75369:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1524.407549] LNet: Removed LNI 192.168.204.52@tcp [ 1525.479262] LNet: Removed LNI 192.168.204.52@tcp1 [ 1525.507854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.608200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1525.874550] LNet: 75414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.249396] Key type .llcrypt unregistered [ 1526.253955] Key type ._llcrypt unregistered [ 1534.116184] Key type ._llcrypt registered [ 1534.120573] Key type .llcrypt registered [ 1535.093454] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:51:46 (1782481906) [ 1535.431449] LNet: 76184:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1535.431449] mask usage: [+|-] ... [ 1535.579184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1535.598589] alg: No test for adler32 (adler32-zlib) [ 1536.488721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.591518] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1539.013159] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:51:49 (1782481909) [ 1539.093268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1539.118199] LNet: 76508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1539.137490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.180131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1539.195800] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1539.695455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1539.713205] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1540.280668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1540.307539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp1 [ 1540.335752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp1 [ 1544.540343] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:51:55 (1782481915) [ 1546.042675] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg452-server [ 1547.269689] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:51:58 (1782481918) [ 1548.463360] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg452-server [ 1549.980568] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:52:00 (1782481920) [ 1551.356602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg452-server [ 1552.485908] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:52:03 (1782481923) [ 1553.706944] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg452-server [ 1554.877695] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:52:06 (1782481926) [ 1556.206826] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg452-server [ 1557.615328] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:52:08 (1782481928) [ 1559.093769] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg452-server [ 1560.346407] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:52:11 (1782481931) [ 1561.599474] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg452-server [ 1562.706416] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:52:13 (1782481933) [ 1564.047894] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg452-server [ 1565.248680] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 09:52:16 (1782481936) [ 1566.588524] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg452-server [ 1567.828753] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:52:18 (1782481938) [ 1567.974154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.993040] LNet: 80386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1568.003564] LNetError: 80386:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1568.009989] LNet: Removed LNI 192.168.204.52@tcp [ 1569.061473] LNet: Removed LNI 192.168.204.52@tcp1 [ 1569.076967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.123738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1569.140413] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1569.715281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1569.780133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1569.935579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.956114] LNet: 80533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.962968] LNetError: 80533:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.968230] LNetError: 80533:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1570.986229] LNet: Removed LNI 192.168.204.52@tcp [ 1571.042810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.120318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1571.151078] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1571.867053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1573.120810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.135037] LNet: 80688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1573.140458] LNetError: 80688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.154723] LNet: Removed LNI 192.168.204.52@tcp [ 1573.179944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.236576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1573.277265] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1573.838717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1573.890919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1574.040518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.078585] LNet: Removed LNI 192.168.204.52@tcp [ 1575.096200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.137931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1575.160036] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1575.676752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1575.885675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.902341] LNetError: 80980:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.906062] LNetError: 80980:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1575.925182] LNet: Removed LNI 192.168.204.52@tcp [ 1575.938700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.975678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1576.002178] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1576.467516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1576.523819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1576.691832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.766415] LNet: Removed LNI 192.168.204.52@tcp [ 1577.780230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.829706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1577.853368] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1578.368203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1578.562239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.578659] LNet: 81273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.583639] LNet: 81273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1579.621836] LNet: Removed LNI 192.168.204.52@tcp [ 1579.665960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.723439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1579.747180] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1580.301251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1580.358696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1580.488274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1580.501791] LNetError: 81420:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1580.506253] LNetError: 81420:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1580.516786] LNet: Removed LNI 192.168.204.52@tcp [ 1580.533602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.569761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1580.599346] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1581.028622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1581.154850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.183915] LNet: Removed LNI 192.168.204.52@tcp [ 1582.205414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.253487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1582.280912] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1582.809694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1582.868859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1584.133516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.190057] LNet: Removed LNI 192.168.204.52@tcp [ 1585.213907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.266322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1585.292677] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1585.761914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1585.906962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.984803] LNet: Removed LNI 192.168.204.52@tcp [ 1587.021162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.069751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1587.089620] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1587.583295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1587.636683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1587.846385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.859582] LNet: 82015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.865856] LNet: 82015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1588.901471] LNet: Removed LNI 192.168.204.52@tcp [ 1588.916298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.961362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1588.989986] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1589.491725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1589.682869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.708311] LNetError: 82160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.712935] LNetError: 82160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1590.759946] LNet: Removed LNI 192.168.204.52@tcp [ 1590.780693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.820136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1590.848745] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1591.307443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1591.355161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.377675] LNet: Removed LNI 192.168.204.52@tcp [ 1591.392684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.428855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1591.453277] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1591.957403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1596.282602] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:52:47 (1782481967) [ 1596.353055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.372941] LNet: Removed LNI 192.168.204.52@tcp [ 1596.388575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.435677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1596.455857] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 1597.285876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.310838] LNet: Removed LNI 192.168.204.52@tcp231 [ 1597.329879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.370352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1597.385098] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 1597.390047] LNet: 83143:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1597.689426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1597.718070] LNet: Removed LNI 192.168.204.52@tcp231 [ 1597.734898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1597.758136] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 1598.527079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.566299] LNet: Removed LNI 192.168.204.52@tcp231 [ 1598.583055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.641336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1598.660406] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/0] [ 1598.670432] LNet: 83295:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1599.012853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1599.047864] LNet: Removed LNI 192.168.204.52@tcp231 [ 1599.058730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1599.086219] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/0] [ 1599.938794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1599.963909] LNet: Removed LNI 192.168.204.52@tcp231 [ 1599.980373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.034182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1600.057477] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/60] [ 1600.060697] LNet: 83446:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1600.336713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1600.364231] LNet: Removed LNI 192.168.204.52@tcp231 [ 1600.378576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1600.399845] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/60] [ 1601.264776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.301761] LNet: Removed LNI 192.168.204.52@tcp231 [ 1601.317391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.385446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1601.405793] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 1601.834804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1601.862988] LNet: Removed LNI 192.168.204.52@tcp231 [ 1601.885921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1601.922095] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 1606.767329] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:52:57 (1782481977) [ 1606.924397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1606.947151] LNet: 84307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1606.952256] LNet: 84307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 1606.958917] LNetError: 84307:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1606.962979] LNetError: 84307:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1606.973794] LNet: Removed LNI 192.168.204.52@tcp231 [ 1606.995547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.044943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1607.075112] LNet: Added LNI 192.168.204.52@tcp232 [8/256/0/180] [ 1607.537708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1607.571703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1607.604804] LNet: Removed LNI 192.168.204.52@tcp232 [ 1607.624506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1607.649463] LNet: Added LNI 192.168.204.52@tcp232 [8/256/0/180] [ 1608.122940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1608.160215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1608.188192] LNet: Removed LNI 192.168.204.52@tcp232 [ 1608.204082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1608.227370] LNet: Added LNI 192.168.204.52@tcp232 [8/256/0/180] [ 1608.682982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1608.714340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1608.744703] LNet: Removed LNI 192.168.204.52@tcp232 [ 1608.757381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1608.783562] LNet: Added LNI 192.168.204.52@tcp232 [8/256/0/180] [ 1609.280360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1609.316658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1609.343700] LNet: Removed LNI 192.168.204.52@tcp232 [ 1609.359110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1613.420889] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 09:53:04 (1782481984) [ 1614.091771] Key type .llcrypt unregistered [ 1614.097122] Key type ._llcrypt unregistered [ 1618.023728] Key type ._llcrypt registered [ 1618.026640] Key type .llcrypt registered [ 1618.079232] LNet: 85663:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1618.079232] mask usage: [+|-] ... [ 1618.230224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1618.244190] alg: No test for adler32 (adler32-zlib) [ 1619.095787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1619.117597] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1624.250757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1629.916632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.960261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1629.978082] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1630.870843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_interrupt -r 2 [ 1630.911214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_interrupt -r 2 [ 1630.934465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_interrupt -r 2 [ 1630.961749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_interrupt -r 2 [ 1630.992474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_interrupt -r 2 [ 1631.016498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_interrupt -r 2 [ 1631.046982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_interrupt -r 2 [ 1631.081814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_interrupt -r 2 [ 1631.112419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1631.128379] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1631.131951] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1631.297331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1631.321820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1631.347912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1631.375439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1631.414347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1631.443068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1631.470918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1631.501567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1631.518981] LNet: 86095:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1631.534464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1631.556462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1631.577474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1631.600254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1631.624264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1631.649974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1631.994782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_dropped -r 2 [ 1632.018687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_dropped -r 2 [ 1632.046258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_dropped -r 2 [ 1632.069548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_dropped -r 2 [ 1632.092368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_dropped -r 2 [ 1632.121528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_dropped -r 2 [ 1632.145521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_dropped -r 2 [ 1632.178141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_dropped -r 2 [ 1632.205146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1632.223511] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1632.378427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1632.395379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1632.412788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1632.432952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1632.454776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1632.481472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1632.506930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1632.538309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1632.553673] LNet: 86216:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1632.568266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1632.593758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1632.620775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1632.658953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1632.691270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1632.722323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1633.084122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_aborted -r 2 [ 1633.118958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_aborted -r 2 [ 1633.146917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_aborted -r 2 [ 1633.178554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_aborted -r 2 [ 1633.231625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_aborted -r 2 [ 1633.258795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_aborted -r 2 [ 1633.285157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_aborted -r 2 [ 1633.317601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_aborted -r 2 [ 1633.344260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1633.361455] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1633.543287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1633.565903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1633.585747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1633.607428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1633.634721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1633.657828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1633.681294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1633.708860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1633.726552] LNet: 86336:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1633.739856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1633.764323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1633.788530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1633.823617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1633.851827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1633.875563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1634.183983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_no_route -r 2 [ 1634.208476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_no_route -r 2 [ 1634.237276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_no_route -r 2 [ 1634.271876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_no_route -r 2 [ 1634.306598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_no_route -r 2 [ 1634.335715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_no_route -r 2 [ 1634.371626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_no_route -r 2 [ 1634.398954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_no_route -r 2 [ 1634.427480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1634.587545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1634.606573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1634.634239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1634.661513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1634.682024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1634.707453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1634.736473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1634.759539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1634.779454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1634.803504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1634.835655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1634.859447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1634.878647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1634.915315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1635.217771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_timeout -r 2 [ 1635.252292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_timeout -r 2 [ 1635.286499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_timeout -r 2 [ 1635.317864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_timeout -r 2 [ 1635.351497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_timeout -r 2 [ 1635.383630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_timeout -r 2 [ 1635.422251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_timeout -r 2 [ 1635.458365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_timeout -r 2 [ 1635.492629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1635.510215] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1635.514758] LNet: Skipped 1 previous similar message [ 1635.680496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1635.699767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1635.732359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1635.758920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1635.795442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1635.824247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1635.857843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1635.884654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1635.899390] LNet: 86576:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1635.909036] LNet: 86576:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1635.929942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1635.954362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1635.992833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1636.018369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1636.039678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1636.060953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1636.392727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_dropped -r 2 [ 1636.428117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e remote_dropped -r 2 [ 1636.454430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_dropped -r 2 [ 1636.477580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e remote_dropped -r 2 [ 1636.513306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e remote_dropped -r 2 [ 1636.551084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e remote_dropped -r 2 [ 1636.599665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e remote_dropped -r 2 [ 1636.629495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e remote_dropped -r 2 [ 1636.659679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1636.680579] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.152@tcp [ 1636.817685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1636.845992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1636.872686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1636.907110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1636.930403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1636.954347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1636.977440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1637.006476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1637.038594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1637.065220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1637.090185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1637.116538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1637.138481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1637.162860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1640.362709] LNet: 86770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1640.368820] LNetError: 86770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1640.375807] LNet: Removed LNI 192.168.204.52@tcp [ 1640.389572] LNet: Removed LNI 192.168.204.52@tcp1 [ 1640.637362] Key type .llcrypt unregistered [ 1640.639718] Key type ._llcrypt unregistered [ 1649.092811] Key type ._llcrypt registered [ 1649.094939] Key type .llcrypt registered [ 1649.926296] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 09:53:41 (1782482021) [ 1650.117721] LNet: 87575:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1650.117721] mask usage: [+|-] ... [ 1650.234499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1650.249181] alg: No test for adler32 (adler32-zlib) [ 1651.091737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.128317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1651.146034] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1651.713797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1651.740658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.52@tcp [ 1651.758930] LNetError: 87747:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.52@tcp to recovery [ 1651.815871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.52@tcp [ 1651.892138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1655.798886] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 09:53:47 (1782482027) [ 1655.871787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.887636] LNet: 88332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.892419] LNetError: 88332:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.904364] LNet: Removed LNI 192.168.204.52@tcp [ 1655.919501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.958488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1655.975849] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1656.403775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1656.419791] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1656.945664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.52@tcp [ 1656.959640] LNetError: 88602:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.52@tcp to recovery [ 1656.992312] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1656.994417] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1658.017524] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1658.117478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.52@tcp [ 1665.971566] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 09:53:57 (1782482037) [ 1667.028142] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg452-server [ 1668.141647] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 09:53:59 (1782482039) [ 1669.348776] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg452-server [ 1670.370506] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 09:54:01 (1782482041) [ 1671.459539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg452-server [ 1672.488979] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 09:54:03 (1782482043) [ 1672.642950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1672.657919] LNet: 90249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.663718] LNetError: 90249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.668824] LNet: Removed LNI 192.168.204.52@tcp [ 1672.680124] LNet: Removed LNI 192.168.204.52@tcp1 [ 1672.694114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.731056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1672.750889] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1673.247925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1673.285976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1673.317517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1673.369602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1673.413104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1673.465908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1678.228164] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 09:54:09 (1782482049) [ 1678.319650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.344161] LNet: 90969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1678.352991] LNetError: 90969:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.358659] LNetError: 90969:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1678.376498] LNet: Removed LNI 192.168.204.52@tcp [ 1678.402936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.458650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1678.482331] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1683.361903] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 09:54:14 (1782482054) [ 1683.475882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.501575] LNet: 91677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.515373] LNetError: 91677:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.529838] LNet: Removed LNI 192.168.204.52@tcp [ 1683.549801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.592614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1683.610092] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1688.072211] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 09:54:19 (1782482059) [ 1688.942279] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1689.910134] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 09:54:21 (1782482061) [ 1690.150788] LNet: 92712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.157082] LNetError: 92712:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.167039] LNet: Removed LNI 192.168.204.52@tcp [ 1690.510364] Key type .llcrypt unregistered [ 1690.512178] Key type ._llcrypt unregistered [ 1693.940767] Key type ._llcrypt registered [ 1693.943483] Key type .llcrypt registered [ 1693.965731] LNet: 92940:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1693.965731] mask usage: [+|-] ... [ 1694.100306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1694.116507] alg: No test for adler32 (adler32-zlib) [ 1694.959121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1694.977571] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1699.788150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1702.657809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.152@tcp [ 1702.902125] LNet: 93127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.908613] LNetError: 93127:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.929468] LNet: Removed LNI 192.168.204.52@tcp [ 1703.146296] Key type .llcrypt unregistered [ 1703.151470] Key type ._llcrypt unregistered [ 1710.211128] Key type ._llcrypt registered [ 1710.213301] Key type .llcrypt registered [ 1711.079892] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1712.146234] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1713.078362] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 09:54:44 (1782482084) [ 1714.096607] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1715.207348] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 09:54:46 (1782482086) [ 1716.300264] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg452-server [ 1717.436983] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 09:54:48 (1782482088) [ 1718.811725] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg452-server [ 1720.001323] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 09:54:51 (1782482091) [ 1720.658845] LNet: 95178:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1720.658845] mask usage: [+|-] ... [ 1720.882700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1720.893507] alg: No test for adler32 (adler32-zlib) [ 1721.849572] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 1721.993483] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1723.624432] Key type lgssc registered [ 1724.434672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1736.863814] Key type lgssc unregistered [ 1737.023857] LNet: 96298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1737.027551] LNetError: 96298:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1737.043670] LNet: Removed LNI 192.168.204.52@tcp [ 1737.483465] Key type .llcrypt unregistered [ 1737.487661] Key type ._llcrypt unregistered [ 1746.699894] Key type ._llcrypt registered [ 1746.703325] Key type .llcrypt registered [ 1747.563323] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 09:55:18 (1782482118) [ 1747.726584] LNet: 97094:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1747.726584] mask usage: [+|-] ... [ 1747.839776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1747.850542] alg: No test for adler32 (adler32-zlib) [ 1748.711464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.769633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1748.820240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1748.863472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1748.891210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1752.733495] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 09:55:23 (1782482123) [ 1752.882517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.901829] LNet: 97711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.926243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.969631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1752.991097] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1753.667222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.681119] LNet: 97855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.685900] LNetError: 97855:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1753.699639] LNet: Removed LNI 192.168.204.52@tcp [ 1753.714864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.766051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1753.790139] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1754.081518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1754.100823] LNet: Removed LNI 192.168.204.52@tcp [ 1754.113547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.170154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1754.192193] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1758.417200] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 09:55:29 (1782482129) [ 1758.539659] LNet: 98473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1758.543643] LNet: 98473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1758.547776] LNetError: 98473:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1758.559166] LNetError: 98473:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1758.575847] LNet: Removed LNI 192.168.204.52@tcp [ 1759.011392] Key type .llcrypt unregistered [ 1759.015675] Key type ._llcrypt unregistered [ 1762.893491] Key type ._llcrypt registered [ 1762.895515] Key type .llcrypt registered [ 1762.926613] LNet: 98706:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1762.926613] mask usage: [+|-] ... [ 1763.028545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1763.040408] alg: No test for adler32 (adler32-zlib) [ 1763.899995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.957491] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1763.961719] LNet: *** cfs_fail_loc=e003, val=100*** [ 1764.451812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1764.468944] LNet: *** cfs_fail_loc=e003, val=100*** [ 1764.471717] LNet: Skipped 2 previous similar messages [ 1764.476030] Ping reply received with mapping: [ 1764.477307] found metadata reply NI[192.168.204.52@tcp] [ 1764.480470] 34 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1764.483338] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1764.486299] 5a 5a 5a 5a 5a 5a 5a 5a [ 1764.566955] LNet: 98872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.571503] LNetError: 98872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.579474] LNet: Removed LNI 192.168.204.52@tcp [ 1764.967311] Key type .llcrypt unregistered [ 1764.971055] Key type ._llcrypt unregistered [ 1771.934656] Key type ._llcrypt registered [ 1771.938298] Key type .llcrypt registered [ 1772.897732] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 09:55:43 (1782482143) [ 1773.277536] Key type .llcrypt unregistered [ 1773.278975] Key type ._llcrypt unregistered [ 1777.773763] Key type ._llcrypt registered [ 1777.775443] Key type .llcrypt registered [ 1777.814772] LNet: 99867:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1777.814772] mask usage: [+|-] ... [ 1777.955665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1777.972769] alg: No test for adler32 (adler32-zlib) [ 1779.018468] LNetError: 99917:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1779.331895] Key type .llcrypt unregistered [ 1779.334475] Key type ._llcrypt unregistered [ 1782.854326] Key type ._llcrypt registered [ 1782.856798] Key type .llcrypt registered [ 1783.860337] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 09:55:54 (1782482154) [ 1784.233829] Key type .llcrypt unregistered [ 1784.237366] Key type ._llcrypt unregistered [ 1788.675739] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1788.682197] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1791.096374] Key type ._llcrypt registered [ 1791.098370] Key type .llcrypt registered [ 1791.121752] LNet: 100785:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1791.121752] mask usage: [+|-] ... [ 1791.218172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1791.227587] alg: No test for adler32 (adler32-zlib) [ 1792.106232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.165212] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1798.302979] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 09:56:09 (1782482169) [ 1798.360549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.373735] LNet: 101693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.379652] LNetError: 101693:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1798.392057] LNet: Removed LNI 10.1.2.3@tcp [ 1798.408854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.863590] Key type .llcrypt unregistered [ 1798.866333] Key type ._llcrypt unregistered [ 1802.400518] Key type ._llcrypt registered [ 1802.402404] Key type .llcrypt registered [ 1802.423492] LNet: 101928:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1802.423492] mask usage: [+|-] ... [ 1802.511679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1802.524931] alg: No test for adler32 (adler32-zlib) [ 1803.393760] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1803.440872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1803.556311] LNet: 101985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.561479] LNetError: 101985:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.585257] LNet: Removed LNI 192.168.204.52@tcp [ 1805.107499] Key type .llcrypt unregistered [ 1805.109509] Key type ._llcrypt unregistered [ 1812.697780] Key type ._llcrypt registered [ 1812.699923] Key type .llcrypt registered [ 1813.753404] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 09:56:24 (1782482184) [ 1814.260303] Key type .llcrypt unregistered [ 1814.262477] Key type ._llcrypt unregistered [ 1818.648174] Key type ._llcrypt registered [ 1818.650153] Key type .llcrypt registered [ 1818.679957] LNet: 102968:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1818.679957] mask usage: [+|-] ... [ 1818.846216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1818.865799] alg: No test for adler32 (adler32-zlib) [ 1823.686869] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 09:56:34 (1782482194) [ 1823.762767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1823.804900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.847992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1823.865897] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1824.281788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1824.301535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1824.325139] LNet: Removed LNI 192.168.204.52@tcp [ 1824.483225] LNet: 103728:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1824.783476] Key type .llcrypt unregistered [ 1824.785921] Key type ._llcrypt unregistered [ 1831.878604] Key type ._llcrypt registered [ 1831.881379] Key type .llcrypt registered [ 1832.929936] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 09:56:43 (1782482203) [ 1833.119525] LNet: 104492:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1833.119525] mask usage: [+|-] ... [ 1833.231674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1833.243077] alg: No test for adler32 (adler32-zlib) [ 1834.184589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.244137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1834.265641] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1834.779788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1834.799907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.52@tcp [ 1834.814587] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1834.817457] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1834.820096] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1834, 0, 0 [ 1834.827577] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1834.829575] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1834.832562] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1834.835553] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1834, 0, 0 [ 1834.843257] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1834.845358] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1834.848471] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1834.851872] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1834, 0, 1884 [ 1834.855477] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1834.857298] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1834.859795] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1834.862623] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1834, 0, 1884 [ 1834.866914] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1840.002816] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 09:56:51 (1782482211) [ 1840.440258] LNet: 105266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.447659] LNetError: 105266:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.464192] LNet: Removed LNI 192.168.204.52@tcp [ 1840.906278] Key type .llcrypt unregistered [ 1840.914032] Key type ._llcrypt unregistered [ 1844.993483] Key type ._llcrypt registered [ 1844.997496] Key type .llcrypt registered [ 1845.028267] LNet: 105489:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1845.028267] mask usage: [+|-] ... [ 1845.150654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1845.160274] alg: No test for adler32 (adler32-zlib) [ 1846.051410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1846.072656] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1851.436929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1857.409897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.442076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1857.461714] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1858.214076] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1858.313723] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1858.316859] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1858.319606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1860.586095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1860.608705] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1868.201726] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:57:19 (1782482239) [ 1868.286475] LNet: 106768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1868.293545] LNetError: 106768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1868.301142] LNet: Removed LNI 192.168.204.52@tcp [ 1869.343168] LNet: Removed LNI 192.168.204.52@tcp1 [ 1869.346930] LNetError: 106768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1869.355344] LNetError: 106768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1869.380483] LNet: Removed LNI 10.1.2.3@tcp99 [ 1869.837518] Key type .llcrypt unregistered [ 1869.841568] Key type ._llcrypt unregistered [ 1873.854578] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1873.859038] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1876.338720] Key type ._llcrypt registered [ 1876.340637] Key type .llcrypt registered [ 1876.366378] LNet: 107036:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1876.366378] mask usage: [+|-] ... [ 1876.476319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1876.486026] alg: No test for adler32 (adler32-zlib) [ 1877.363646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1877.403353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1877.436736] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1878.022735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1878.040206] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1879.641994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1879.677241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1879.717987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1879.734567] LNetError: 107514:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1879.753177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1879.781543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1880.083966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1880.142728] LNet: 107534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.151099] LNetError: 107534:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1880.156979] LNet: Removed LNI 10.1.2.3@tcp [ 1881.191779] LNet: Removed LNI 192.168.204.52@tcp [ 1881.683166] Key type .llcrypt unregistered [ 1881.691751] Key type ._llcrypt unregistered [ 1889.427935] Key type ._llcrypt registered [ 1889.431786] Key type .llcrypt registered [ 1890.333473] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:57:41 (1782482261) [ 1890.530508] LNet: 108332:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1890.530508] mask usage: [+|-] ... [ 1890.619484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1890.630255] alg: No test for adler32 (adler32-zlib) [ 1891.523470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1891.573706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1891.593511] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1892.151879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1892.208945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg452-client@tcp [ 1896.564359] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:57:47 (1782482267) [ 1896.717430] LNet: 109065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.727690] LNetError: 109065:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.748447] LNet: Removed LNI 192.168.204.52@tcp [ 1897.274606] Key type .llcrypt unregistered [ 1897.278460] Key type ._llcrypt unregistered [ 1902.104352] Key type ._llcrypt registered [ 1902.108162] Key type .llcrypt registered [ 1902.268147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1902.282718] alg: No test for adler32 (adler32-zlib) [ 1903.144138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1903.168577] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1903.192593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1907.880857] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:57:58 (1782482278) [ 1908.823575] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1909.804320] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:58:01 (1782482281) [ 1910.958330] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1912.181772] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 09:58:03 (1782482283) [ 1912.233102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1912.250506] LNet: 110479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1912.255539] LNetError: 110479:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1912.266075] LNet: Removed LNI 192.168.204.52@tcp [ 1912.290748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1912.356188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1912.374273] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1912.849378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1912.885568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1917.484467] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 09:58:08 (1782482288) [ 1917.611978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1917.634996] LNet: 111194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.641543] LNetError: 111194:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1917.654027] LNet: Removed LNI 192.168.204.52@tcp [ 1917.670461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1917.726985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1917.763520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1918.196643] Key type .llcrypt unregistered [ 1918.199726] Key type ._llcrypt unregistered [ 1925.461472] Key type ._llcrypt registered [ 1925.463635] Key type .llcrypt registered [ 1926.687796] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 09:58:17 (1782482297) [ 1926.884528] LNet: 112000:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1926.884528] mask usage: [+|-] ... [ 1927.011400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1927.036907] alg: No test for adler32 (adler32-zlib) [ 1927.973985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1928.009916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1928.030133] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1928.583781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1928.637139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1933.398444] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 09:58:24 (1782482304) [ 1933.462586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1933.478285] LNet: 112745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1933.485469] LNetError: 112745:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1933.497604] LNet: Removed LNI 192.168.204.52@tcp [ 1933.516644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1933.563694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1933.593148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1938.293628] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 09:58:29 (1782482309) [ 1938.368640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1938.382278] LNet: 113324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.399674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.469270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1938.815989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1938.842550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1938.892520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1939.164673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1939.203498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1939.282268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1939.313826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1939.374086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1939.698596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1939.723160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1939.768477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1940.099541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1940.130049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1944.619769] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 09:58:35 (1782482315) [ 1944.772621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1944.793682] LNet: 113963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.799930] LNet: 113963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1944.818822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1944.872700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1944.889215] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1945.393906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1945.407959] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 1946.024959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1946.064221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1946.229292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1946.246959] LNetError: 114242:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1946.257798] LNet: Removed LNI 192.168.204.52@tcp [ 1946.272966] LNet: Removed LNI 192.168.204.52@tcp2 [ 1946.285178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.336031] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1946.340905] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 1946.347566] LNet: 114248:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1946.659450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1946.819685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1946.840177] LNet: 114285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1946.845657] LNet: 114285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1947.871153] LNet: Removed LNI 192.168.204.52@tcp [ 1947.873974] LNetError: 114285:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1947.882734] LNetError: 114285:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1947.896901] LNet: Removed LNI 192.168.204.52@tcp2 [ 1947.917803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1947.974989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1947.992698] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1948.485200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1948.504996] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 1949.176924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1949.218209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1949.268504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1949.448471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1950.495168] LNet: Removed LNI 192.168.204.52@tcp [ 1950.502000] LNetError: 114569:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1950.505801] LNetError: 114569:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1950.532198] LNet: Removed LNI 192.168.204.52@tcp2 [ 1950.560577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1950.612465] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1950.618478] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 1950.622585] LNet: 114575:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1951.103509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1951.320893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1951.340437] LNet: 114613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1951.352462] LNet: 114613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1951.361168] LNet: Removed LNI 192.168.204.52@tcp [ 1952.433139] LNet: Removed LNI 192.168.204.52@tcp2 [ 1952.481296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1952.547381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1952.573656] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1953.143890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1953.162728] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 1953.889508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1953.947667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1954.159633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1955.232347] LNet: Removed LNI 192.168.204.52@tcp [ 1955.237107] LNetError: 114892:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1955.248081] LNetError: 114892:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1955.277942] LNet: Removed LNI 192.168.204.52@tcp2 [ 1955.323646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1955.382240] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1955.388913] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 1955.393126] LNet: 114898:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1955.813348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1955.994470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1957.023232] LNet: Removed LNI 192.168.204.52@tcp [ 1957.031680] LNet: Removed LNI 192.168.204.52@tcp2 [ 1957.057328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1957.114344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1957.141199] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1957.666380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1957.685452] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 1958.359190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1958.388450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1958.430644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1958.570230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1959.647324] LNet: Removed LNI 192.168.204.52@tcp [ 1960.679276] LNet: Removed LNI 192.168.204.52@tcp2 [ 1960.707971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1960.754582] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1960.758743] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 1960.763570] LNet: 115226:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1961.135781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1961.263760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1961.283563] LNet: 115264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1961.288239] LNet: 115264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1961.294846] LNet: Removed LNI 192.168.204.52@tcp [ 1962.344086] LNet: Removed LNI 192.168.204.52@tcp2 [ 1962.364817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.420915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1962.454307] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1962.950682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1962.972690] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 1963.576190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1963.603709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1963.660965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1963.920941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1963.943077] LNetError: 115560:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.949589] LNetError: 115560:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1963.956818] LNet: Removed LNI 192.168.204.52@tcp [ 1963.967907] LNet: Removed LNI 192.168.204.52@tcp2 [ 1963.997730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1964.051125] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1964.055977] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 1964.447447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1964.700118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.727213] LNet: Removed LNI 192.168.204.52@tcp [ 1965.746147] LNet: Removed LNI 192.168.204.52@tcp2 [ 1965.789791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.882872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1965.910406] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1966.538280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1966.557723] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 1967.270303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1967.309107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1967.332382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1967.368365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1967.611560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1968.673099] LNet: Removed LNI 192.168.204.52@tcp [ 1969.702767] LNet: Removed LNI 192.168.204.52@tcp2 [ 1969.719769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1969.781601] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1969.790698] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 1969.794546] LNet: 115926:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1969.800279] LNet: 115926:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1970.170078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1975.376054] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1976.846629] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 09:59:07 (1782482347) [ 1976.977515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1977.005043] LNet: Removed LNI 192.168.204.52@tcp [ 1978.022407] LNet: Removed LNI 192.168.204.52@tcp2 [ 1978.049761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1978.315348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1978.341785] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1978.816392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1978.837687] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1979.474515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1979.493247] LNet: 116958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1979.497291] LNet: 116958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1979.502863] LNet: Removed LNI 10.1.2.3@tcp [ 1979.512577] LNet: Removed LNI 192.168.204.52@tcp [ 1979.527933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1979.583682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1979.613350] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1979.619867] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1979.990894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1980.016552] LNet: Removed LNI 192.168.204.52@tcp [ 1980.032080] LNet: Removed LNI 10.1.2.3@tcp [ 1985.583594] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 09:59:16 (1782482356) [ 1985.710444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1985.754576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1985.833679] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1985.956061] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1985.961156] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1985.965213] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1988.230364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1988.255239] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1988.754258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1988.771697] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1990.276151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1990.310432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1990.386809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.52@tcp -m PUT -e local_timeout -r 1 [ 1990.446526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1990.497807] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1990.501290] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1990.700174] LNetError: 117623:0:(lib-move.c:4714:lnet_parse()) 192.168.204.52@tcp, src 192.168.204.52@tcp: Dropping PUT (error -108 looking up sender) [ 1991.712367] LNetError: 118082:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1991.717900] LNetError: 118082:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 1991.728492] LNet: Removed LNI 10.1.2.3@tcp [ 1992.741404] LNet: Removed LNI 192.168.204.52@tcp [ 1993.093211] Key type .llcrypt unregistered [ 1993.095736] Key type ._llcrypt unregistered [ 2001.133213] Key type ._llcrypt registered [ 2001.139558] Key type .llcrypt registered [ 2002.189281] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 09:59:33 (1782482373) [ 2002.364282] LNet: 118852:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2002.364282] mask usage: [+|-] ... [ 2002.512660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2002.530449] alg: No test for adler32 (adler32-zlib) [ 2003.407277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2003.683707] LNet: 118933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2004.258878] Key type .llcrypt unregistered [ 2004.263452] Key type ._llcrypt unregistered [ 2008.853920] Key type ._llcrypt registered [ 2008.856157] Key type .llcrypt registered [ 2008.896441] LNet: 119162:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2008.896441] mask usage: [+|-] ... [ 2009.053312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2009.074284] alg: No test for adler32 (adler32-zlib) [ 2009.941088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2009.975349] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2015.924596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 2016.091591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2016.580763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.152@tcp [ 2016.790701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2016.818854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2016.870429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2016.901472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2016.925728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2016.959211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2017.006349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2017.047682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2017.074877] LNet: 119383:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2017.088773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2017.112125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2017.139972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2017.179634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2017.183574] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.152@tcp [ 2017.222186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2017.254316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2017.797424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2017.973640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2017.999340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2018.024040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2018.050584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2018.078392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2018.102491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2018.128808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2018.159273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2018.176639] LNet: 119500:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2018.191448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2018.219901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2018.252379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2018.283639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2018.313180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2018.348420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2018.868674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.152@tcp [ 2019.059740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2019.083643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2019.105736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2019.139830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2019.176893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2019.222581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2019.251718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2019.275865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2019.290708] LNet: 119617:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2019.306407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2019.334415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2019.361693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2019.391346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2019.417560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2019.446890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2019.967388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2020.131798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2020.155207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2020.184500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2020.211247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2020.242968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2020.264548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2020.288848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2020.313330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2020.335759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2020.363642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2020.382683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2020.405930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2020.435708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2020.477176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2020.705779] LNet: 119780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2020.715196] LNetError: 119780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2020.735100] LNet: Removed LNI 192.168.204.52@tcp [ 2021.067393] Key type .llcrypt unregistered [ 2021.071276] Key type ._llcrypt unregistered [ 2033.387565] Key type ._llcrypt registered [ 2033.390084] Key type .llcrypt registered [ 2034.422527] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:00:05 (1782482405) [ 2034.590679] LNet: 120583:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2034.590679] mask usage: [+|-] ... [ 2034.702375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2034.712490] alg: No test for adler32 (adler32-zlib) [ 2035.582624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2035.901184] LNet: 120666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.355420] Key type .llcrypt unregistered [ 2036.357374] Key type ._llcrypt unregistered [ 2040.499708] Key type ._llcrypt registered [ 2040.502789] Key type .llcrypt registered [ 2040.534586] LNet: 120895:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2040.534586] mask usage: [+|-] ... [ 2040.675817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2040.686846] alg: No test for adler32 (adler32-zlib) [ 2041.539423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2041.566638] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2047.477306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 2053.726787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2053.758724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2053.778612] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 2054.459138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2054.882624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.152@tcp [ 2055.091888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2055.126198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2055.151271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2055.175806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2055.201712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2055.224621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2055.255870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2055.280819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2055.299100] LNet: 121317:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2055.315914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2055.346584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2055.376828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2055.406805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2055.434031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2055.461204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2055.946954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2056.141196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2056.164809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2056.192186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2056.225590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2056.247428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2056.270923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2056.300436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2056.332577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2056.354300] LNet: 121434:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2056.379616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2056.401991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2056.430542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2056.458196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2056.498119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2056.532375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2057.044323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.152@tcp [ 2057.231529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2057.256878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2057.280169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2057.311776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2057.331826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2057.355719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2057.386621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2057.415795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2057.440326] LNet: 121552:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2057.459509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2057.493841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2057.527936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2057.565744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2057.597352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2057.631625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2058.064710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2058.265271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2058.292609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2058.323400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2058.349032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2058.377848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2058.402272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2058.437828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2058.470372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2058.507448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2058.533478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2058.558562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2058.583318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2058.606818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2058.634647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2062.533535] LNet: 121749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2062.540863] LNetError: 121749:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2062.546729] LNet: Removed LNI 192.168.204.52@tcp [ 2062.557785] LNet: Removed LNI 192.168.204.52@tcp1 [ 2062.867791] Key type .llcrypt unregistered [ 2062.869926] Key type ._llcrypt unregistered [ 2074.415736] Key type ._llcrypt registered [ 2074.418542] Key type .llcrypt registered [ 2075.389906] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 10:00:46 (1782482446) [ 2080.464391] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 10:00:51 (1782482451) [ 2085.484071] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 10:00:56 (1782482456) [ 2090.621481] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 10:01:01 (1782482461) [ 2095.158498] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 10:01:06 (1782482466) [ 2099.836818] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 10:01:11 (1782482471) [ 2104.433785] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 10:01:15 (1782482475) [ 2108.902814] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 10:01:20 (1782482480) [ 2113.898571] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 10:01:24 (1782482484) [ 2118.499960] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 10:01:29 (1782482489) [ 2123.107685] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 10:01:34 (1782482494) [ 2127.783216] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 10:01:38 (1782482498) [ 2132.070586] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 10:01:43 (1782482503) [ 2136.720844] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 10:01:47 (1782482507) [ 2141.847286] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 10:01:52 (1782482512) [ 2146.869982] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 10:01:58 (1782482518) [ 2151.525626] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 10:02:02 (1782482522) [ 2156.258554] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 10:02:07 (1782482527) [ 2156.450382] LNet: 132297:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2156.450382] mask usage: [+|-] ... [ 2156.587410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2156.606699] alg: No test for adler32 (adler32-zlib) [ 2157.512606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.607777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2157.628955] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2158.456671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2158.476450] LNet: 132468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.481711] LNetError: 132468:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2158.494440] LNet: Removed LNI 192.168.204.52@tcp [ 2158.509425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.549988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2158.572576] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2162.852257] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 10:02:14 (1782482534) [ 2167.176358] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 10:02:18 (1782482538) [ 2171.496746] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 10:02:22 (1782482542) [ 2176.464924] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:02:27 (1782482547) [ 2181.138734] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:02:32 (1782482552) [ 2185.495296] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:02:36 (1782482556) [ 2189.728787] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:02:40 (1782482560) [ 2194.146590] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:02:45 (1782482565) [ 2198.615842] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:02:49 (1782482569) [ 2203.327905] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:02:54 (1782482574) [ 2208.704612] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:02:59 (1782482579) [ 2213.300753] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:03:04 (1782482584) [ 2217.530893] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1915 sec ========== 10:03:08 (1782482588) [ 2217.804530] LNet: 139929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2217.810444] LNetError: 139929:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2217.820814] LNet: Removed LNI 192.168.204.52@tcp [ 2218.315897] Key type .llcrypt unregistered [ 2218.322250] Key type ._llcrypt unregistered [ 2227.084495] Key type ._llcrypt registered [ 2227.086830] Key type .llcrypt registered [ 2227.133137] LNet: 140384:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2227.133137] mask usage: [+|-] ... [ 2227.411934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2227.423951] alg: No test for adler32 (adler32-zlib) [ 2228.431257] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 2228.643368] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2230.359239] Key type lgssc registered [ 2231.249339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2278.855411] Lustre: Mounted lustre-client [ 2278.948670] LNet: 141870:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2278.948670] mask usage: [+|-] ... [ 2283.091674] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2289.057972] Lustre: Unmounted lustre-client [ 2308.887089] Key type lgssc unregistered [ 2309.088678] LNet: 142850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.098579] LNetError: 142850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.115940] LNet: Removed LNI 192.168.204.52@tcp [ 2309.631170] Key type .llcrypt unregistered [ 2309.632652] Key type ._llcrypt unregistered