[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489795928 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003208] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.007786] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008044] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009024] pid_max: default: 32768 minimum: 301 [ 0.010183] LSM: Security Framework initializing [ 0.011078] Yama: becoming mindful. [ 0.013029] SELinux: Initializing. [ 0.014094] *** VALIDATE selinux *** [ 0.022772] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027442] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028175] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029130] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030156] *** VALIDATE tmpfs *** [ 0.031530] *** VALIDATE proc *** [ 0.033229] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.036228] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037191] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038019] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040017] Spectre V2 : User space: Vulnerable [ 0.041017] Speculative Store Bypass: Vulnerable [ 0.044503] debug: unmapping init [mem 0xffffffff93259000-0xffffffff93260fff] [ 0.047228] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048850] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049042] ... version: 2 [ 0.050022] ... bit width: 48 [ 0.051021] ... generic registers: 4 [ 0.052018] ... value mask: 0000ffffffffffff [ 0.053025] ... max period: 00007fffffffffff [ 0.054024] ... fixed-purpose events: 3 [ 0.055017] ... event mask: 000000070000000f [ 0.056412] rcu: Hierarchical SRCU implementation. [ 0.058660] smp: Bringing up secondary CPUs ... [ 0.059690] x86: Booting SMP configuration: [ 0.060041] .... node #0, CPUs: #1 #2 #3 [ 0.068014] smp: Brought up 1 node, 4 CPUs [ 0.070019] smpboot: Max logical packages: 1 [ 0.071025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.192917] node 0 deferred pages initialised in 118ms [ 0.197194] devtmpfs: initialized [ 0.198341] x86/mm: Memory block size: 128MB [ 0.200792] gcov: version magic: 0x41383552 [ 0.202267] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.203096] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.204341] pinctrl core: initialized pinctrl subsystem [ 0.205233] [ 0.205836] ************************************************************* [ 0.206018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.207018] ** ** [ 0.208017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.209018] ** ** [ 0.210016] ** This means that this kernel is built to expose internal ** [ 0.211013] ** IOMMU data structures, which may compromise security on ** [ 0.212013] ** your system. ** [ 0.213014] ** ** [ 0.214013] ** If you see this message and you are not debugging the ** [ 0.215014] ** kernel, report this immediately to your vendor! ** [ 0.216012] ** ** [ 0.217012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.218021] ************************************************************* [ 0.219852] NET: Registered protocol family 16 [ 0.220573] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.221096] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.222102] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.223965] cpuidle: using governor menu [ 0.225000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.228710] PCI: Using configuration type 1 for base access [ 0.230144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.239216] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.242198] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.247087] cryptd: max_cpu_qlen set to 1000 [ 0.249616] ACPI: Added _OSI(Module Device) [ 0.251024] ACPI: Added _OSI(Processor Device) [ 0.252085] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.254016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.259363] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.266205] ACPI: Interpreter enabled [ 0.267396] ACPI: PM: (supports S0 S3 S4 S5) [ 0.268021] ACPI: Using IOAPIC for interrupt routing [ 0.269569] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.271000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.280787] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.281069] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.282032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.283105] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.285606] acpiphp: Slot [2] registered [ 0.286242] acpiphp: Slot [5] registered [ 0.287203] acpiphp: Slot [6] registered [ 0.288191] acpiphp: Slot [3] registered [ 0.289144] acpiphp: Slot [4] registered [ 0.290214] acpiphp: Slot [7] registered [ 0.291153] acpiphp: Slot [8] registered [ 0.292113] acpiphp: Slot [9] registered [ 0.293136] acpiphp: Slot [10] registered [ 0.294225] acpiphp: Slot [11] registered [ 0.295132] acpiphp: Slot [12] registered [ 0.296176] acpiphp: Slot [13] registered [ 0.297000] acpiphp: Slot [14] registered [ 0.299138] acpiphp: Slot [15] registered [ 0.301121] acpiphp: Slot [16] registered [ 0.302176] acpiphp: Slot [17] registered [ 0.304139] acpiphp: Slot [18] registered [ 0.306213] acpiphp: Slot [19] registered [ 0.308131] acpiphp: Slot [20] registered [ 0.309143] acpiphp: Slot [21] registered [ 0.311129] acpiphp: Slot [22] registered [ 0.313108] acpiphp: Slot [23] registered [ 0.314091] acpiphp: Slot [24] registered [ 0.316123] acpiphp: Slot [25] registered [ 0.317235] acpiphp: Slot [26] registered [ 0.319236] acpiphp: Slot [27] registered [ 0.321115] acpiphp: Slot [28] registered [ 0.323111] acpiphp: Slot [29] registered [ 0.325097] acpiphp: Slot [30] registered [ 0.326107] acpiphp: Slot [31] registered [ 0.328077] PCI host bridge to bus 0000:00 [ 0.329022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.331027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.334032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.337312] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.342030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.345029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.347257] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.349070] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352274] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.358533] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.362058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.366021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.367016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.369021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.373356] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.376881] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.379044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.381979] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.387018] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.395000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.402034] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.411017] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.433021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.453021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.468346] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.483020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.496020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.513021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.556360] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.560226] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.563453] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.565396] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.567314] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.573117] iommu: Default domain type: Passthrough [ 0.575300] SCSI subsystem initialized [ 0.576283] ACPI: bus type USB registered [ 0.578122] usbcore: registered new interface driver usbfs [ 0.580110] usbcore: registered new interface driver hub [ 0.582111] usbcore: registered new device driver usb [ 0.585187] pps_core: LinuxPPS API ver. 1 registered [ 0.587017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.591098] PTP clock support registered [ 0.594066] EDAC MC: Ver: 3.0.0 [ 0.596147] PCI: Using ACPI for IRQ routing [ 0.599210] NetLabel: Initializing [ 0.600012] NetLabel: domain hash size = 128 [ 0.602020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.604282] NetLabel: unlabeled traffic allowed by default [ 0.607431] vgaarb: loaded [ 0.609524] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.611023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.627440] clocksource: Switched to clocksource kvm-clock [ 0.755359] VFS: Disk quotas dquot_6.6.0 [ 0.757303] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.760314] *** VALIDATE ramfs *** [ 0.762081] *** VALIDATE hugetlbfs *** [ 0.764095] pnp: PnP ACPI init [ 0.766877] pnp: PnP ACPI: found 6 devices [ 0.784193] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.788411] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.791191] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.793846] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.796598] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.799764] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.803466] NET: Registered protocol family 2 [ 0.806397] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.811706] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.816182] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.822473] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.826585] TCP: Hash tables configured (established 65536 bind 65536) [ 0.830145] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.833821] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.837532] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.841231] NET: Registered protocol family 1 [ 0.844290] RPC: Registered named UNIX socket transport module. [ 0.846778] RPC: Registered udp transport module. [ 0.848754] RPC: Registered tcp transport module. [ 0.850854] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.853718] NET: Registered protocol family 44 [ 0.855818] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.858459] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.861139] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.863903] PCI: CLS 0 bytes, default 64 [ 0.865871] Unpacking initramfs... [ 2.634324] debug: unmapping init [mem 0xffff9c473cc64000-0xffff9c473ffcffff] [ 2.644214] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.647601] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.652826] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.215689] Initialise system trusted keyrings [ 3.217666] Key type blacklist registered [ 3.219790] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.229957] zbud: loaded [ 3.247992] *** VALIDATE nfs *** [ 3.249469] *** VALIDATE nfs4 *** [ 3.255439] pstore: using deflate compression [ 3.262369] Platform Keyring initialized [ 3.423728] NET: Registered protocol family 38 [ 3.427385] Key type asymmetric registered [ 3.429616] Asymmetric key parser 'x509' registered [ 3.432603] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.437387] io scheduler mq-deadline registered [ 3.439420] io scheduler kyber registered [ 3.443470] io scheduler bfq registered [ 3.445816] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.452767] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.456601] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.460272] ACPI: Power Button [PWRF] [ 3.469200] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.479536] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.576127] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.620499] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.674866] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.680737] Non-volatile memory driver v1.3 [ 3.682944] Linux agpgart interface v0.103 [ 3.725370] virtio_blk virtio1: [vda] 145176 512-byte logical blocks (74.3 MB/70.9 MiB) [ 3.728893] vda: detected capacity change from 0 to 74330112 [ 3.754518] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.759354] vdb: detected capacity change from 0 to 1073741824 [ 3.770956] libphy: Fixed MDIO Bus: probed [ 3.780142] usbcore: registered new interface driver usbserial_generic [ 3.782847] usbserial: USB Serial support registered for generic [ 3.785561] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.790950] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.793125] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.795765] mousedev: PS/2 mouse device common for all mice [ 3.798393] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.801899] rtc_cmos 00:05: RTC can wake from S4 [ 3.810105] rtc_cmos 00:05: registered as rtc0 [ 3.813716] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.816557] intel_pstate: CPU model not supported [ 3.820493] hid: raw HID events driver (C) Jiri Kosina [ 3.822520] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.825115] usbcore: registered new interface driver usbhid [ 3.833245] usbhid: USB HID core driver [ 3.835198] drop_monitor: Initializing network drop monitor service [ 3.837052] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.840016] Initializing XFRM netlink socket [ 3.848392] NET: Registered protocol family 10 [ 3.851319] Segment Routing with IPv6 [ 3.852923] NET: Registered protocol family 17 [ 3.855521] mpls_gso: MPLS GSO support [ 3.861187] RAS: Correctable Errors collector initialized. [ 3.863427] AVX version of gcm_enc/dec engaged. [ 3.864937] AES CTR mode by8 optimization enabled [ 3.954780] sched_clock: Marking stable (3954686828, 0)->(4942959584, -988272756) [ 3.959558] registered taskstats version 1 [ 3.965249] Loading compiled-in X.509 certificates [ 3.968156] zswap: loaded using pool lzo/zbud [ 4.001210] Key type big_key registered [ 4.019178] Key type encrypted registered [ 4.020799] ima: No TPM chip found, activating TPM-bypass! [ 4.023561] ima: Allocated hash algorithm: sha1 [ 4.025344] ima: No architecture policies found [ 4.027491] evm: Initialising EVM extended attributes: [ 4.029760] evm: security.selinux [ 4.031421] evm: security.ima [ 4.032661] evm: security.capability [ 4.034469] evm: HMAC attrs: 0x1 [ 4.036595] rtc_cmos 00:05: setting system clock to 2026-07-07 16:16:48 UTC (1783441008) [ 4.043829] debug: unmapping init [mem 0xffffffff94203000-0xffffffff943fffff] [ 4.048147] debug: unmapping init [mem 0xffffffff92f82000-0xffffffff93258fff] [ 4.057372] Write protecting the kernel read-only data: 28672k [ 4.062797] debug: unmapping init [mem 0xffffffff91603000-0xffffffff917fffff] [ 4.066612] debug: unmapping init [mem 0xffffffff91f14000-0xffffffff91ffffff] [ 4.109677] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.119250] systemd[1]: Detected virtualization kvm. [ 4.120987] systemd[1]: Detected architecture x86-64. [ 4.123532] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.154948] systemd[1]: No hostname configured. [ 4.156409] systemd[1]: Set hostname to . [ 4.158248] random: systemd: uninitialized urandom read (16 bytes read) [ 4.160286] systemd[1]: Initializing machine ID from random generator. [ 4.251469] random: ln: uninitialized urandom read (6 bytes read) [ 4.581365] random: systemd: uninitialized urandom read (16 bytes read) [ 4.584368] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.589684] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.595108] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.372541] device-mapper: uevent: version 1.0.3 [ 6.375565] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ 7.993043] hrtimer: interrupt took 8126709 ns Mounting Kernel Configuration File System... [ 8.196380] virtio_net virtio0 ens2: renamed from eth0 [ 8.711252] scsi host0: ata_piix [ 8.719059] scsi host1: ata_piix [ 8.722852] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 8.731018] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ OK ] Mounted Kernel Configuration File System. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 14.865819] random: crng init done [ 14.871500] random: 7 urandom warning(s) missed due to ratelimiting [ 16.462625] dracut-initqueue[571]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.109919] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.491881] printk: systemd: 19 output lines suppressed due to ratelimiting [ 22.523856] SELinux: Disabled at runtime. [ 22.672337] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.701137] systemd[1]: Detected virtualization kvm. [ 22.711188] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.070149] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.088231] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.117473] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.123314] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.127345] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.157421] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.190812] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Me[ 26.032359] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ssage Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 27.061787] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.101063] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.200041] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.454465] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.607837] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 35.231194] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit)[ 36.815271] NFS: Registering the id_resolver key type [ 36.821961] Key type id_resolver registered [ 36.832098] Key type id_legacy registered [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg408-client login: [ 105.548450] libcfs: loading out-of-tree module taints kernel. [ 105.749559] Key type ._llcrypt registered [ 105.761476] Key type .llcrypt registered [ 106.860796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 106.888502] alg: No test for adler32 (adler32-zlib) [ 108.593583] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 109.567807] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 111.449743] Key type lgssc registered [ 113.472994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 243.850844] Lustre: Mounted lustre-client [ 248.934738] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 265.237271] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing check_logdir /tmp/testlogs/ [ 269.280749] Lustre: lustre-OST0000-osc-ffff9c47891be000: disconnect after 22s idle [ 271.524526] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing yml_node [ 275.925610] Lustre: DEBUG MARKER: Client: 2.17.54.105 [ 278.958955] Lustre: DEBUG MARKER: MDS: 2.17.54.105 [ 281.743789] Lustre: DEBUG MARKER: OSS: 2.17.54.105 [ 283.555549] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Jul 7 12:21:26 EDT 2026 [ 299.697240] Lustre: Unmounted lustre-client [ 326.563576] Key type lgssc unregistered [ 326.996451] LNet: 8393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.013404] LNetError: 8393:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 327.058864] LNet: Removed LNI 192.168.204.8@tcp [ 327.989149] Key type .llcrypt unregistered [ 328.001117] Key type ._llcrypt unregistered [ 358.532817] Key type ._llcrypt registered [ 358.538272] Key type .llcrypt registered [ 358.584274] LNet: 8972:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 358.584274] mask usage: [+|-] ... [ 358.789517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 358.806460] alg: No test for adler32 (adler32-zlib) [ 359.672245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 359.731066] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 359.745200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 364.101554] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 364.306610] LNet: 9275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.324288] LNetError: 9275:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 364.346699] LNet: Removed LNI 192.168.204.8@tcp [ 364.958305] Key type .llcrypt unregistered [ 364.961754] Key type ._llcrypt unregistered [ 371.641356] Key type ._llcrypt registered [ 371.647039] Key type .llcrypt registered [ 373.353785] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:22:55 (1783441375) [ 373.763593] LNet: 9651:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 373.763593] mask usage: [+|-] ... [ 373.990266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 374.022266] alg: No test for adler32 (adler32-zlib) [ 375.009138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.106459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.133298] LNet: 9689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.159297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.251181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.280607] LNet: 9696:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 383.127956] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:23:05 (1783441385) [ 383.248463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.284286] LNet: 10270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.308101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.378297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 390.506705] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:23:13 (1783441393) [ 390.660840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.688280] LNet: 10862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 390.711649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.782697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 397.500767] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:23:20 (1783441400) [ 397.760594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.788297] LNet: 11446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.829674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.934710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 405.683430] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:23:28 (1783441408) [ 405.781128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.837442] LNet: 12024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 405.889785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.030351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 406.487053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 413.791381] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:23:36 (1783441416) [ 413.997797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.041143] LNet: 12619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.071666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.197721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 414.673983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 415.334628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 423.068187] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:23:45 (1783441425) [ 423.289779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.387487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.501404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 430.104219] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:23:53 (1783441433) [ 430.235824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.268631] LNet: 13798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 430.275406] LNet: 13798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 430.306455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.369852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 430.396908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 430.736431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 430.764728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 431.157266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 431.203624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 431.821062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 431.924222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 432.431569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 432.481456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 432.857630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 432.899260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 433.311919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 433.358394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 433.885394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 433.924950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 434.392761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 434.450081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 434.913841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 434.960223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 435.665950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 435.724661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 436.325680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 436.389272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 436.937574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 436.991246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 443.751652] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:24:06 (1783441446) [ 443.985555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.039575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.175729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 444.246623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 452.465326] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:24:15 (1783441455) [ 452.800912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.877389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.994518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 453.039541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 461.388067] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:24:24 (1783441464) [ 461.504502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.578332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.756910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 461.813887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 469.956992] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:24:32 (1783441472) [ 470.336383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.361581] LNet: 16244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 470.370027] LNet: 16244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 470.419724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.535648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 470.573369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 477.722393] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:24:40 (1783441480) [ 477.857419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.925832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.060775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 478.119453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 484.538748] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:24:47 (1783441487) [ 484.937844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.979505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.076890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 485.126937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 492.827700] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:24:55 (1783441495) [ 493.189360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.261507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.393887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 493.960436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 501.116193] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:25:04 (1783441504) [ 501.484695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.541596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.731835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 501.999246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 509.435886] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:25:12 (1783441512) [ 509.672701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.704729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.852351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.113926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 517.097378] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:25:20 (1783441520) [ 517.227790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.301476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.970474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 518.259817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 524.816453] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:25:28 (1783441528) [ 525.030600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.071723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.136425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 525.172196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 531.818240] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:25:34 (1783441534) [ 531.968813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.043177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.186384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 532.250141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 540.077275] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:25:43 (1783441543) [ 540.280245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.318585] LNet: 21687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 540.324189] LNet: 21687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 540.348580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.421344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 547.524739] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:25:50 (1783441550) [ 547.672309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.747351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.843337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 554.700863] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:25:57 (1783441557) [ 554.929773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 555.029975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.126039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 555.730929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 562.511663] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:26:05 (1783441565) [ 562.641091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.694986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.840500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 562.887894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 570.409702] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:26:13 (1783441573) [ 570.554280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.578241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.696478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 570.778445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 579.257686] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:26:21 (1783441581) [ 579.379928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.441261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.546669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 579.649529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 589.235500] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:26:31 (1783441591) [ 589.655309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.693938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.805680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 589.841270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 599.321445] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:26:42 (1783441602) [ 599.512562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.583994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.733218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 599.830376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 599.847758] LNetError: 25806:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 599.895590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 599.944678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 606.308070] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:26:49 (1783441609) [ 606.525434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.559277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.629721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.4d2RSu8jPm [ 606.724600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.4d2RSu8jPm [ 606.768626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.4d2RSu8jPm [ 612.699874] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:26:55 (1783441615) [ 612.827406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.881837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.967967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 613.031050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 613.090791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 619.988107] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:27:03 (1783441623) [ 620.296907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.341683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.413484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 623.062419] LNet: Message forwarding will be enabled in 2s [ 626.146918] LNet: Message forwarding enabled [ 626.635594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 626.653529] LNet: Message forwarding will stop in 2s to 6s [ 631.331812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 632.007764] LNet: Message forwarding disabled [ 634.099248] LNet: Message forwarding will be enabled in 2s [ 638.176629] LNet: Message forwarding enabled [ 641.278169] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:27:24 (1783441644) [ 641.398168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.893428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.966930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 648.369690] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:27:31 (1783441651) [ 648.449444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.507816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.596245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 648.638639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 648.674142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 648.715720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 648.777318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 648.825781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 648.881119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 648.933325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 648.988367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 649.051863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 649.114271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 649.166833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 649.217217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 649.252614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 649.288244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 649.321281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 649.357820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 649.396671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 649.439778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 649.464833] LNetError: 28878:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 655.203592] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:27:38 (1783441658) [ 655.284390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.320713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.421991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 661.299629] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:27:44 (1783441664) [ 663.102930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg408-server [ 665.038972] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:27:47 (1783441667) [ 665.120577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.170103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.240075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 665.270863] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 666.771406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[9-13]@tcp [ 666.796802] LNet: 30593:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 666.847339] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.9@tcp is being used as a gateway but routing feature is not turned on [ 666.877566] LNetError: 30380:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.9@tcp has gone from up to down [ 667.879174] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.9@tcp is being used as a gateway but routing feature is not turned on [ 667.894357] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 668.906283] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.10@tcp is being used as a gateway but routing feature is not turned on [ 668.922906] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 670.954722] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.10@tcp is being used as a gateway but routing feature is not turned on [ 670.979963] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 672.838855] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:27:56 (1783441676) [ 674.515672] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg408-server [ 675.052226] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.13@tcp is being used as a gateway but routing feature is not turned on [ 675.064927] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 676.322725] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:27:59 (1783441679) [ 676.526162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.552136] LNet: 31527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.559070] LNet: 31527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 676.566969] LNetError: 31527:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.609757] LNet: Removed LNI 192.168.204.8@tcp [ 676.657124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.742808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 676.792646] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 677.558342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[9-14/2]@tcp [ 677.583491] LNet: 31645:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 677.589547] LNet: 31645:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 677.609356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[9-14/2]@tcp [ 677.620131] LNetError: 31532:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.13@tcp has gone from up to down [ 677.631682] LNetError: 31532:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 684.971128] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:28:07 (1783441687) [ 685.092521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.124053] LNetError: 32233:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.143227] LNet: Removed LNI 192.168.204.8@tcp [ 685.195600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.266839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 685.309593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.337808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.385722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 685.429416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.472381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.525311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 685.634754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.658197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.717107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 685.802807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.854569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.900799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 685.975814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.003592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.060570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 686.153888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.201654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.255769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 686.361937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.407436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.471376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 686.559409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.589891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.640715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 686.760825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.807513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.899299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 686.975531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.006844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.048590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 687.099379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.130832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.192558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 694.931885] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:28:17 (1783441697) [ 696.532235] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg408-server [ 698.072304] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:28:21 (1783441701) [ 700.747981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg408-server [ 702.827908] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:28:25 (1783441705) [ 704.029497] Key type .llcrypt unregistered [ 704.032589] Key type ._llcrypt unregistered [ 710.731361] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 710.935223] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 710.940724] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 710.952651] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 713.489949] Key type ._llcrypt registered [ 713.492712] Key type .llcrypt registered [ 713.538324] LNet: 33921:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 713.538324] mask usage: [+|-] ... [ 713.741332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 713.764626] alg: No test for adler32 (adler32-zlib) [ 714.692932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.790311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 714.830085] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 715.743317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 715.788623] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 717.858991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 717.883914] LNet: Removed LNI 10.1.2.3@tcp [ 717.989279] LNet: 34363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.995222] LNetError: 34363:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.017888] LNet: Removed LNI 192.168.204.8@tcp [ 718.570295] Key type .llcrypt unregistered [ 718.573313] Key type ._llcrypt unregistered [ 730.439154] Key type ._llcrypt registered [ 730.443746] Key type .llcrypt registered [ 732.209459] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:28:54 (1783441734) [ 732.695779] Key type .llcrypt unregistered [ 732.700530] Key type ._llcrypt unregistered [ 739.703961] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 739.810860] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 739.819531] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 739.833293] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 742.771266] Key type ._llcrypt registered [ 742.774846] Key type .llcrypt registered [ 742.826262] LNet: 35399:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 742.826262] mask usage: [+|-] ... [ 742.989093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 743.015845] alg: No test for adler32 (adler32-zlib) [ 744.025765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.111336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 744.143169] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 745.142916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 745.174404] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 748.182811] LNet: 35856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.188260] LNetError: 35856:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 748.199289] LNet: Removed LNI 10.1.2.3@tcp [ 748.216578] LNet: Removed LNI 192.168.204.8@tcp [ 748.957370] Key type .llcrypt unregistered [ 748.961655] Key type ._llcrypt unregistered [ 763.196691] Key type ._llcrypt registered [ 763.201761] Key type .llcrypt registered [ 764.489545] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:29:27 (1783441767) [ 765.098169] Key type .llcrypt unregistered [ 765.104470] Key type ._llcrypt unregistered [ 771.882110] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 771.976976] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 771.985879] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 771.993702] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 774.698418] Key type ._llcrypt registered [ 774.704061] Key type .llcrypt registered [ 774.757578] LNet: 36905:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 774.757578] mask usage: [+|-] ... [ 774.958837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 774.976238] alg: No test for adler32 (adler32-zlib) [ 775.978536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.080912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 776.142586] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 778.717863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 778.748719] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 781.327460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 781.353567] LNet: Removed LNI 10.1.2.3@tcp [ 781.383475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 781.421462] LNet: Removed LNI 10.1.2.31@tcp [ 781.439485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 781.481517] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 782.253683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 782.271783] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 783.441718] LNet: 37789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.450547] LNetError: 37789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 783.457362] LNet: Removed LNI 10.1.2.31@tcp [ 783.481994] LNet: Removed LNI 10.1.2.3@tcp [ 784.289261] Key type .llcrypt unregistered [ 784.292789] Key type ._llcrypt unregistered [ 797.612949] Key type ._llcrypt registered [ 797.616515] Key type .llcrypt registered [ 798.990659] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:30:01 (1783441801) [ 799.350741] LNet: 38582:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 799.350741] mask usage: [+|-] ... [ 799.538443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 799.563246] alg: No test for adler32 (adler32-zlib) [ 800.575963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.666216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 800.724814] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 801.612206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.640868] LNet: 38742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.646949] LNetError: 38742:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 801.661686] LNet: Removed LNI 192.168.204.8@tcp [ 801.698377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.787216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.8@tcp [ 801.813625] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 802.635525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.8@tcp [ 802.665553] LNet: Removed LNI 192.168.204.8@tcp [ 802.767850] LNet: 38878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.546778] Key type .llcrypt unregistered [ 803.560329] Key type ._llcrypt unregistered [ 816.191363] Key type ._llcrypt registered [ 816.197298] Key type .llcrypt registered [ 817.555825] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:30:20 (1783441820) [ 819.884977] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg408-server [ 821.708905] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:30:24 (1783441824) [ 822.369339] Key type .llcrypt unregistered [ 822.373819] Key type ._llcrypt unregistered [ 828.951735] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 829.036331] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 829.049176] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 829.062224] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 831.851200] Key type ._llcrypt registered [ 831.855577] Key type .llcrypt registered [ 831.909590] LNet: 40254:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 831.909590] mask usage: [+|-] ... [ 832.150658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 832.182631] alg: No test for adler32 (adler32-zlib) [ 833.172154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.242037] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 833.982494] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 836.381883] LNet: 40714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.389885] LNetError: 40714:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 836.395592] LNet: Removed LNI 192.168.204.8@tcp1 [ 836.411927] LNet: Removed LNI 10.1.2.3@tcp2 [ 837.172329] Key type .llcrypt unregistered [ 837.179183] Key type ._llcrypt unregistered [ 849.191497] Key type ._llcrypt registered [ 849.194723] Key type .llcrypt registered [ 850.500514] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:30:53 (1783441853) [ 850.823239] LNet: 41518:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 850.823239] mask usage: [+|-] ... [ 850.970785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 850.986612] alg: No test for adler32 (adler32-zlib) [ 851.928561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.017715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 852.049240] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 852.822921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.845725] LNet: 41683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.853505] LNetError: 41683:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.873744] LNet: Removed LNI 192.168.204.8@tcp [ 852.897341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.957830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 852.980128] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 860.038843] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:31:03 (1783441863) [ 860.131643] LNet: 42266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.143566] LNetError: 42266:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.165126] LNet: Removed LNI 192.168.204.8@tcp [ 860.832531] Key type .llcrypt unregistered [ 860.838412] Key type ._llcrypt unregistered [ 867.794814] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 868.109757] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 868.124807] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 868.145870] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 870.942090] Key type ._llcrypt registered [ 870.950332] Key type .llcrypt registered [ 871.016214] LNet: 42551:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 871.016214] mask usage: [+|-] ... [ 871.323399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 871.356597] alg: No test for adler32 (adler32-zlib) [ 872.379124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.472632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 872.516410] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 873.630222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 873.686044] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 876.910742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.943228] LNet: 43017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.954056] LNetError: 43017:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 876.960107] LNet: Removed LNI 10.1.2.3@tcp [ 876.974908] LNet: Removed LNI 192.168.204.8@tcp [ 877.012307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.119342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 877.152679] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 877.157164] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 877.685552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.714606] LNet: 43044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.724726] LNetError: 43044:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 877.731705] LNetError: 43044:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 877.741065] LNet: Removed LNI 10.1.2.3@tcp [ 877.755898] LNet: Removed LNI 192.168.204.8@tcp [ 877.799964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.856414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 877.896253] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 877.901807] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 878.434859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 878.481223] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 879.658325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.686065] LNet: 43207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.695106] LNetError: 43207:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 879.703935] LNetError: 43207:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 879.730126] LNet: Removed LNI 10.1.2.3@tcp [ 879.739123] LNet: Removed LNI 192.168.204.8@tcp [ 879.752553] LNet: Removed LNI 192.168.204.8@tcp2 [ 879.786350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.861924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 879.902513] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 879.906131] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 879.911818] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 888.674099] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:31:31 (1783441891) [ 888.831309] LNet: 43798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.848410] LNetError: 43798:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 888.870543] LNetError: 43798:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 888.895560] LNet: Removed LNI 10.1.2.3@tcp [ 888.897804] LNet: Removed LNI 192.168.204.8@tcp [ 888.921330] LNet: Removed LNI 192.168.204.8@tcp2 [ 889.789923] Key type .llcrypt unregistered [ 889.792247] Key type ._llcrypt unregistered [ 897.751236] Key type ._llcrypt registered [ 897.755761] Key type .llcrypt registered [ 897.798270] LNet: 44030:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 897.798270] mask usage: [+|-] ... [ 898.086445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 898.115188] alg: No test for adler32 (adler32-zlib) [ 899.126962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.222487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 899.254524] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 900.243995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.261764] LNet: 44200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.270162] LNetError: 44200:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.296173] LNet: Removed LNI 192.168.204.8@tcp [ 900.349817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.411702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 900.434033] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 900.908370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.955191] LNet: 44230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.983914] LNetError: 44230:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.008813] LNet: Removed LNI 192.168.204.8@tcp [ 901.040634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.130598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 901.171517] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 901.832986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.882310] LNet: Removed LNI 192.168.204.8@tcp [ 901.902705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.971845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 902.030234] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 902.804322] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 902.937991] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 902.948064] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 902.951704] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 905.545460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.573665] LNet: 44342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.578948] LNet: 44342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 905.585794] LNetError: 44342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.593975] LNetError: 44342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 905.614512] LNet: Removed LNI 192.168.204.8@tcp [ 905.634259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.727261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 905.756650] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 913.317173] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:31:55 (1783441915) [ 913.495566] LNet: 44931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.509848] LNetError: 44931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.529856] LNet: Removed LNI 192.168.204.8@tcp [ 914.701340] Key type .llcrypt unregistered [ 914.705760] Key type ._llcrypt unregistered [ 922.548624] Key type ._llcrypt registered [ 922.567437] Key type .llcrypt registered [ 922.668059] LNet: 45165:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 922.668059] mask usage: [+|-] ... [ 922.879315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 922.894824] alg: No test for adler32 (adler32-zlib) [ 923.834499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.919877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 923.945099] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 924.750574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.774583] LNet: 45338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.782556] LNetError: 45338:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.805493] LNet: Removed LNI 192.168.204.8@tcp [ 924.848289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.923260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 924.943188] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 931.873478] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:32:14 (1783441934) [ 932.065819] LNet: 45931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.075268] LNetError: 45931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.093345] LNet: Removed LNI 192.168.204.8@tcp [ 932.783439] Key type .llcrypt unregistered [ 932.787839] Key type ._llcrypt unregistered [ 940.034622] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 940.281881] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 940.296161] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 940.305449] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 943.161914] Key type ._llcrypt registered [ 943.167770] Key type .llcrypt registered [ 943.226171] LNet: 46207:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 943.226171] mask usage: [+|-] ... [ 943.544779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 943.570687] alg: No test for adler32 (adler32-zlib) [ 944.521569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.598672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 944.625531] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 945.613594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.642450] LNet: 46381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.650588] LNetError: 46381:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.663859] LNet: Removed LNI 192.168.204.8@tcp [ 945.680545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.759473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 945.783644] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 953.267256] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:32:36 (1783441956) [ 953.384273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.443575] LNet: 46975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.451255] LNetError: 46975:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.479297] LNet: Removed LNI 192.168.204.8@tcp [ 953.527984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.640378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 960.410082] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:32:43 (1783441963) [ 960.493675] LNet: 47552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.219772] Key type .llcrypt unregistered [ 961.223736] Key type ._llcrypt unregistered [ 968.235198] Key type ._llcrypt registered [ 968.237951] Key type .llcrypt registered [ 968.295708] LNet: 47787:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 968.295708] mask usage: [+|-] ... [ 968.558561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 968.580153] alg: No test for adler32 (adler32-zlib) [ 969.548710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.641807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 969.688063] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 970.542383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.585353] LNet: 47959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.601914] LNetError: 47959:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.619834] LNet: Removed LNI 192.168.204.8@tcp [ 970.648452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.741759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 970.765077] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 977.735340] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:33:00 (1783441980) [ 977.804534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 977.829465] LNet: 48546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.832706] LNetError: 48546:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.849128] LNet: Removed LNI 192.168.204.8@tcp [ 977.866903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.949598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 983.564254] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:33:06 (1783441986) [ 985.158772] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 986.833946] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:33:10 (1783441990) [ 988.092684] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 989.539492] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:33:12 (1783441992) [ 991.546757] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 993.810602] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:33:16 (1783441996) [ 995.909450] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 997.716471] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:33:20 (1783442000) [ 999.010568] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1001.060754] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:33:23 (1783442003) [ 1001.216243] LNet: 50595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.052408] Key type .llcrypt unregistered [ 1002.058522] Key type ._llcrypt unregistered [ 1008.838296] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1008.918940] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1008.924038] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1008.933344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1011.622193] Key type ._llcrypt registered [ 1011.625170] Key type .llcrypt registered [ 1011.674479] LNet: 50882:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1011.674479] mask usage: [+|-] ... [ 1011.858882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1011.888811] alg: No test for adler32 (adler32-zlib) [ 1012.961240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.081388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1013.108850] LNet: Added LNI 192.168.204.8@tcp [8/512/0/180] [ 1013.118323] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1013.122852] LNet: 50921:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1013.245826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1013.287870] LNet: 50940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.298681] LNetError: 50940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1013.305389] LNet: Removed LNI 10.1.2.3@tcp [ 1013.309152] LNetError: 50940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.325618] LNet: Removed LNI 192.168.204.8@tcp [ 1013.367524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.484435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1013.528435] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1013.649336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1013.686875] LNet: 50964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.692977] LNetError: 50964:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.712609] LNet: Removed LNI 192.168.204.8@tcp [ 1013.768498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.903655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1013.949958] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1013.960312] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1013.968363] LNet: 50972:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1014.077186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1014.097598] LNet: 50990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.104640] LNetError: 50990:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1014.109683] LNet: Removed LNI 10.1.2.3@tcp [ 1014.112382] LNetError: 50990:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.126221] LNet: Removed LNI 192.168.204.8@tcp [ 1014.166741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.237485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1014.261967] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1014.265925] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1014.273141] LNet: 50998:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1014.598718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1014.625378] LNet: 51024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.633487] LNetError: 51024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1014.644654] LNet: Removed LNI 10.1.2.3@tcp [ 1014.653256] LNetError: 51024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.686087] LNet: Removed LNI 192.168.204.8@tcp [ 1014.724176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.799959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1014.826264] LNet: Added LNI 192.168.204.8@tcp [8/256/0/182] [ 1014.833227] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1014.969357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1015.005364] LNet: 51052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.015755] LNetError: 51052:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1015.030932] LNet: Removed LNI 10.1.2.3@tcp [ 1015.042754] LNetError: 51052:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.057965] LNet: Removed LNI 192.168.204.8@tcp [ 1015.075455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.184722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1015.231882] LNet: Added LNI 192.168.204.8@tcp [8/256/0/182] [ 1015.238190] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1015.243311] LNet: 51060:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1015.327023] LNet: Removed LNI 10.1.2.3@tcp [ 1015.348989] LNet: Removed LNI 192.168.204.8@tcp [ 1015.935326] Key type .llcrypt unregistered [ 1015.937668] Key type ._llcrypt unregistered [ 1026.773943] Key type ._llcrypt registered [ 1026.777067] Key type .llcrypt registered [ 1028.023773] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:33:51 (1783442031) [ 1028.321098] LNet: 51876:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1028.321098] mask usage: [+|-] ... [ 1028.515782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1028.541723] alg: No test for adler32 (adler32-zlib) [ 1029.456216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.530540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1029.564934] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1030.279565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.8 [ 1030.294516] LNet: 52037:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1030.307877] LNet: Removed LNI 192.168.204.8@tcp [ 1036.169710] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:33:59 (1783442039) [ 1036.352126] LNet: 52608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.360877] LNetError: 52608:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.378824] LNet: Removed LNI 192.168.204.8@tcp [ 1036.871398] Key type .llcrypt unregistered [ 1036.875687] Key type ._llcrypt unregistered [ 1042.577202] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1042.824392] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1045.475348] Key type ._llcrypt registered [ 1045.480888] Key type .llcrypt registered [ 1045.518336] LNet: 52890:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1045.518336] mask usage: [+|-] ... [ 1045.674979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1045.695131] alg: No test for adler32 (adler32-zlib) [ 1046.954042] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1053.072759] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:34:16 (1783442056) [ 1053.171845] LNet: 53512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.177801] LNetError: 53512:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1053.192103] LNet: Removed LNI 10.1.2.3@tcp [ 1053.732563] Key type .llcrypt unregistered [ 1053.735259] Key type ._llcrypt unregistered [ 1058.337361] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1058.528305] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1058.532755] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1058.540348] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1061.159313] Key type ._llcrypt registered [ 1061.162060] Key type .llcrypt registered [ 1061.197216] LNet: 53795:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1061.197216] mask usage: [+|-] ... [ 1061.298977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1061.322251] alg: No test for adler32 (adler32-zlib) [ 1062.318659] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1068.461193] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:34:31 (1783442071) [ 1070.100385] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1072.178069] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:34:34 (1783442074) [ 1072.446167] LNet: 54705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.483457] LNetError: 54705:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1072.500282] LNet: Removed LNI 10.1.2.3@tcp [ 1073.261756] Key type .llcrypt unregistered [ 1073.265579] Key type ._llcrypt unregistered [ 1081.200043] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1081.473484] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1081.483887] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1081.492394] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1084.381225] Key type ._llcrypt registered [ 1084.388585] Key type .llcrypt registered [ 1084.445312] LNet: 54982:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1084.445312] mask usage: [+|-] ... [ 1084.642764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1084.689725] alg: No test for adler32 (adler32-zlib) [ 1085.732542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1085.805646] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1094.414902] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:34:57 (1783442097) [ 1094.952076] LNet: 55938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1094.965553] LNetError: 55938:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1094.987766] LNet: Removed LNI 10.1.2.3@tcp [ 1095.617428] Key type .llcrypt unregistered [ 1095.621060] Key type ._llcrypt unregistered [ 1101.082625] Key type ._llcrypt registered [ 1101.085074] Key type .llcrypt registered [ 1101.145613] LNet: 56162:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1101.145613] mask usage: [+|-] ... [ 1101.300902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1101.320327] alg: No test for adler32 (adler32-zlib) [ 1102.230691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1102.264440] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1109.688281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1110.271333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_interrupt -r 1 [ 1110.311952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_interrupt -r 1 [ 1110.353523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1110.373615] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1110.496351] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1110.570133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.608258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1110.645454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.674132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.706152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.735821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.764230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.795196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.823121] LNet: 56377:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1110.845595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.882405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.910670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.937832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.971405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1111.007394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.033563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1111.514877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_dropped -r 1 [ 1111.567925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_dropped -r 1 [ 1111.597795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1111.619311] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1111.805461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1111.830816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1111.850243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1111.872043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1111.903854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1111.942637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1111.974750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1112.005456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1112.029855] LNet: 56486:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1112.051148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1112.093214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1112.123747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.147426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.180224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.224719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.253968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1112.665452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_aborted -r 1 [ 1112.702337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_aborted -r 1 [ 1112.734811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1112.763629] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1112.969771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.013583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1113.050974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1113.094912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1113.120705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1113.150977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1113.181770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1113.216575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1113.232585] LNet: 56595:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1113.252139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1113.281844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1113.308947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1113.333546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1113.359372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.385157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1113.408331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1113.835444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_no_route -r 1 [ 1113.870389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_no_route -r 1 [ 1113.902581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1114.095804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1114.117945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1114.138668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.163461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.186807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.208322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.238661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.264383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.289678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.316992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1114.338452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1114.363556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1114.391446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.419634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1114.448747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1114.796172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_timeout -r 1 [ 1114.827285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_timeout -r 1 [ 1114.863625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1114.881976] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1114.885714] LNet: Skipped 1 previous similar message [ 1115.021659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1115.044224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1115.080903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1115.111745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1115.133637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1115.155234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1115.175691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1115.199110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1115.219291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1115.241322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1115.261483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.282719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.308863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.348609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1115.381749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1115.712467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_error -r 1 [ 1115.737675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_error -r 1 [ 1115.761786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1115.912611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1115.934299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1115.973440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1115.998149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.030383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.062430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.092530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.133948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.160684] LNet: 56923:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1116.169721] LNet: 56923:0:(api-ni.c:294:recovery_interval_set()) Skipped 2 previous similar messages [ 1116.191126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.217493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.262991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.291644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1116.318141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.347717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1116.381543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1116.588314] LNet: 56964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1116.595579] LNetError: 56964:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1116.618838] LNet: Removed LNI 192.168.204.8@tcp [ 1116.891531] Key type .llcrypt unregistered [ 1116.894161] Key type ._llcrypt unregistered [ 1129.437092] Key type ._llcrypt registered [ 1129.442952] Key type .llcrypt registered [ 1130.756240] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:35:33 (1783442133) [ 1131.481728] Key type .llcrypt unregistered [ 1131.484374] Key type ._llcrypt unregistered [ 1136.831073] Key type ._llcrypt registered [ 1136.834778] Key type .llcrypt registered [ 1136.866723] LNet: 58002:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1136.866723] mask usage: [+|-] ... [ 1137.015348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1137.028936] alg: No test for adler32 (adler32-zlib) [ 1137.921058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1137.945614] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1144.092069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1152.922840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1152.968153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1152.998956] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1154.100153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_interrupt -r 1 [ 1154.135647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_interrupt -r 1 [ 1154.171383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_interrupt -r 1 [ 1154.207766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_interrupt -r 1 [ 1154.253758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_interrupt -r 1 [ 1154.286992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_interrupt -r 1 [ 1154.319620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_interrupt -r 1 [ 1154.358315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_interrupt -r 1 [ 1154.402683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1154.434206] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1154.442574] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1154.651830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.680444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1154.710722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.732835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.757338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.783678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.805598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1154.829086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1154.856148] LNet: 58431:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1154.890088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1154.922140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1154.956331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1154.986874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.026901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.084827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.119944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1155.474684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_dropped -r 1 [ 1155.513699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_dropped -r 1 [ 1155.544758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_dropped -r 1 [ 1155.582477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_dropped -r 1 [ 1155.611847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_dropped -r 1 [ 1155.637213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_dropped -r 1 [ 1155.669501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_dropped -r 1 [ 1155.694931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_dropped -r 1 [ 1155.720682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1155.734752] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1155.738941] LNet: Skipped 2 previous similar messages [ 1155.920480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1155.944909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1155.973524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1155.997609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1156.025147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1156.054834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1156.090332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1156.121638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1156.145743] LNet: 58552:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1156.160570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1156.200300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1156.230229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1156.256576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1156.298922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1156.345227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1156.383168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1156.837686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_aborted -r 1 [ 1156.862907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_aborted -r 1 [ 1156.895669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_aborted -r 1 [ 1156.919491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_aborted -r 1 [ 1156.953421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_aborted -r 1 [ 1156.990765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_aborted -r 1 [ 1157.028911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_aborted -r 1 [ 1157.063391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_aborted -r 1 [ 1157.110959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1157.135466] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1157.141714] LNet: Skipped 2 previous similar messages [ 1157.350689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1157.378272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1157.414511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1157.454677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1157.480763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1157.510239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1157.554649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1157.580651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1157.601614] LNet: 58674:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1157.628577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1157.659691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1157.704322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1157.759937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1157.828420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1157.870566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1157.910108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1158.446610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_no_route -r 1 [ 1158.483454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_no_route -r 1 [ 1158.525587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_no_route -r 1 [ 1158.582596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_no_route -r 1 [ 1158.634251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_no_route -r 1 [ 1158.673518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_no_route -r 1 [ 1158.721927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_no_route -r 1 [ 1158.758450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_no_route -r 1 [ 1158.819453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1159.094875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1159.118433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1159.158114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1159.202945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1159.247129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1159.286655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1159.326217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1159.369920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1159.395645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1159.440146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1159.464879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1159.492348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1159.528976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.565131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1159.599730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1160.110310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_timeout -r 1 [ 1160.144324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_timeout -r 1 [ 1160.183104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_timeout -r 1 [ 1160.235271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_timeout -r 1 [ 1160.296624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_timeout -r 1 [ 1160.354890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_timeout -r 1 [ 1160.402471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_timeout -r 1 [ 1160.458478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_timeout -r 1 [ 1160.517116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1160.544784] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1160.551207] LNet: Skipped 5 previous similar messages [ 1160.759577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1160.818392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1160.862386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1160.908918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1160.941499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1160.980897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.025143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1161.069890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1161.094925] LNet: 58916:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1161.104887] LNet: 58916:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1161.135675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1161.183261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1161.227522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1161.264419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1161.298489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1161.329551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1161.374762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1161.979317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_error -r 1 [ 1162.046971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_error -r 1 [ 1162.118562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_error -r 1 [ 1162.181150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_error -r 1 [ 1162.230396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_error -r 1 [ 1162.296468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_error -r 1 [ 1162.356835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_error -r 1 [ 1162.412275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_error -r 1 [ 1162.460825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1162.777618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1162.807721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1162.834981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1162.863215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1162.889711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1162.915776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1162.943807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1162.970732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1162.994978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.019775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.050771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.079855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.114293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.141393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.169788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1168.751511] LNet: 59114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.759396] LNetError: 59114:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.772974] LNet: Removed LNI 192.168.204.8@tcp [ 1168.789291] LNet: Removed LNI 192.168.204.8@tcp1 [ 1169.162443] Key type .llcrypt unregistered [ 1169.164913] Key type ._llcrypt unregistered [ 1187.042294] Key type ._llcrypt registered [ 1187.045330] Key type .llcrypt registered [ 1188.681319] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:36:31 (1783442191) [ 1189.606845] Key type .llcrypt unregistered [ 1189.610439] Key type ._llcrypt unregistered [ 1199.291088] Key type ._llcrypt registered [ 1199.294127] Key type .llcrypt registered [ 1199.362336] LNet: 60150:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1199.362336] mask usage: [+|-] ... [ 1199.610794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1199.649269] alg: No test for adler32 (adler32-zlib) [ 1200.632766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1200.664213] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1208.139526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1208.891953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e remote_dropped -r 1 [ 1208.921624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e remote_dropped -r 1 [ 1208.960269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1208.975254] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1209.177880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1209.216561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1209.249253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1209.299606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1209.360452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1209.414626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1209.447331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1209.505676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1209.541503] LNet: 60365:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1209.571741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1209.608384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1209.664578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1209.727471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1209.797237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1209.833379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1209.854613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1210.327086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e remote_error -r 1 [ 1210.371968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e remote_error -r 1 [ 1210.407211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1210.424247] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1210.688916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1210.719633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1210.750833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1210.793946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1210.828230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1210.874828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1210.913738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1210.957202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1210.991867] LNet: 60474:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1211.015179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1211.054813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1211.096191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.134291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1211.179838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1211.233248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1211.275829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1211.994955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e remote_timeout -r 1 [ 1212.023701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e remote_timeout -r 1 [ 1212.076193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1212.124256] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1212.310397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.367480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1212.416852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.465170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1212.529672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1212.559658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1212.595305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1212.649549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.672048] LNet: 60584:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1212.698334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1212.742934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.779020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.828407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.868719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.929231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1212.973329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1213.252171] LNet: 60625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1213.266918] LNetError: 60625:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1213.292207] LNet: Removed LNI 192.168.204.8@tcp [ 1213.857630] Key type .llcrypt unregistered [ 1213.865173] Key type ._llcrypt unregistered [ 1232.113831] Key type ._llcrypt registered [ 1232.118690] Key type .llcrypt registered [ 1233.683518] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:37:16 (1783442236) [ 1234.758071] Key type .llcrypt unregistered [ 1234.761916] Key type ._llcrypt unregistered [ 1240.635360] Key type ._llcrypt registered [ 1240.642815] Key type .llcrypt registered [ 1240.705921] LNet: 61664:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1240.705921] mask usage: [+|-] ... [ 1240.868067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1240.882141] alg: No test for adler32 (adler32-zlib) [ 1241.909586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1241.945822] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1248.966784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1256.750175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1256.790037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1256.808825] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1258.241133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e remote_dropped -r 1 [ 1258.272660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e remote_dropped -r 1 [ 1258.317204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e remote_dropped -r 1 [ 1258.352457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e remote_dropped -r 1 [ 1258.396096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e remote_dropped -r 1 [ 1258.442461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e remote_dropped -r 1 [ 1258.479387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e remote_dropped -r 1 [ 1258.527285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e remote_dropped -r 1 [ 1258.570424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1258.587307] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1258.592855] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.108@tcp [ 1258.827544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1258.861262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1258.902155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1258.943292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1258.978765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1259.009748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1259.060543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1259.100733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1259.122838] LNet: 62093:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1259.155699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1259.181292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1259.235284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1259.297187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1259.351981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1259.387821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1259.412797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1259.447786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1259.997757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e remote_error -r 1 [ 1260.031560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e remote_error -r 1 [ 1260.071472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e remote_error -r 1 [ 1260.116476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e remote_error -r 1 [ 1260.167883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e remote_error -r 1 [ 1260.226425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e remote_error -r 1 [ 1260.278273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e remote_error -r 1 [ 1260.335775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e remote_error -r 1 [ 1260.391356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1260.414697] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1260.422495] LNet: Skipped 2 previous similar messages [ 1260.616768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1260.663639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1260.701365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1260.737791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1260.768878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1260.801266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1260.847949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1260.885804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1260.915944] LNet: 62216:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1260.943864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1260.976311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1261.019465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1261.065804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1261.099465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1261.148562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1261.190872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1261.233607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1261.771881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e remote_timeout -r 1 [ 1261.797858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e remote_timeout -r 1 [ 1261.825877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e remote_timeout -r 1 [ 1261.863600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e remote_timeout -r 1 [ 1261.921371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e remote_timeout -r 1 [ 1261.973728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e remote_timeout -r 1 [ 1262.018468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e remote_timeout -r 1 [ 1262.052584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e remote_timeout -r 1 [ 1262.092341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1262.116853] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1262.328881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1262.377616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1262.434029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1262.470838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1262.525982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1262.570534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1262.607245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1262.652026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1262.668970] LNet: 62339:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1262.687562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1262.736951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1262.801443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1262.838559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1262.869454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1262.919364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1262.960484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1262.997697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1267.388173] LNet: 62417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1267.394287] LNetError: 62417:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1267.406640] LNet: Removed LNI 192.168.204.8@tcp [ 1267.432107] LNet: Removed LNI 192.168.204.8@tcp1 [ 1267.776374] Key type .llcrypt unregistered [ 1267.780137] Key type ._llcrypt unregistered [ 1283.394593] Key type ._llcrypt registered [ 1283.397117] Key type .llcrypt registered [ 1284.701665] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:38:07 (1783442287) [ 1286.670901] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1288.159419] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:38:11 (1783442291) [ 1288.964916] Key type .llcrypt unregistered [ 1288.969946] Key type ._llcrypt unregistered [ 1295.602894] Key type ._llcrypt registered [ 1295.606437] Key type .llcrypt registered [ 1295.660770] LNet: 63746:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1295.660770] mask usage: [+|-] ... [ 1295.853334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1295.877885] alg: No test for adler32 (adler32-zlib) [ 1296.775412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1296.797568] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1304.481631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1305.407426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e network_timeout -r 1 [ 1305.459837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e network_timeout -r 1 [ 1305.500443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1305.531368] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1305.805319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1305.871671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1305.914141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1305.960700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1306.012411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1306.022088] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1306.071795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1306.124464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1306.177774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1306.218099] LNet: 63961:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1306.270675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1306.321153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1306.361439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1306.439860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1306.480746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1306.533318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1306.941529] LNet: 64000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.953825] LNetError: 64000:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.974599] LNet: Removed LNI 192.168.204.8@tcp [ 1307.609371] Key type .llcrypt unregistered [ 1307.613213] Key type ._llcrypt unregistered [ 1326.334858] Key type ._llcrypt registered [ 1326.337570] Key type .llcrypt registered [ 1326.373447] LNet: 64451:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1326.373447] mask usage: [+|-] ... [ 1326.631728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1326.655447] alg: No test for adler32 (adler32-zlib) [ 1327.624995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1327.671784] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1336.181455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1346.481771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1346.524828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1346.554118] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1348.154166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e network_timeout -r 1 [ 1348.190703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e network_timeout -r 1 [ 1348.222640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e network_timeout -r 1 [ 1348.285689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e network_timeout -r 1 [ 1348.339948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e network_timeout -r 1 [ 1348.377946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e network_timeout -r 1 [ 1348.427929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e network_timeout -r 1 [ 1348.477221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e network_timeout -r 1 [ 1348.534828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1348.551822] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1348.846222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1348.885352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1348.924735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1348.977711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1349.027431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1349.083929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1349.135025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1349.201165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1349.217447] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1349.217465] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.108@tcp [ 1349.218108] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1349.247713] LNet: 64881:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1349.249321] LNet: Skipped 1 previous similar message [ 1349.270772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1349.302552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1349.329547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1349.388896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1349.444332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.491309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1354.262445] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1354.613539] LNet: 64955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1354.622881] LNetError: 64955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1354.635408] LNet: Removed LNI 192.168.204.8@tcp [ 1354.648530] LNet: Removed LNI 192.168.204.8@tcp1 [ 1355.119372] Key type .llcrypt unregistered [ 1355.125941] Key type ._llcrypt unregistered [ 1370.006623] Key type ._llcrypt registered [ 1370.008698] Key type .llcrypt registered [ 1371.230147] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:39:34 (1783442374) [ 1371.529261] LNet: 65756:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1371.529261] mask usage: [+|-] ... [ 1371.655519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1371.670450] alg: No test for adler32 (adler32-zlib) [ 1372.550670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1372.592852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1372.616322] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1373.229880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1373.260910] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1374.038467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 1374.093563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1374.128481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1374.155797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1374.209604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1374.260454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1374.328368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.8@tcp [ 1374.372194] LNetError: 66055:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.8@tcp to recovery [ 1374.626781] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1374.633980] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1376.672240] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1377.901468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1380.768790] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1381.455605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1381.513098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1381.587663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1381.618869] LNet: 66162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1381.631262] LNetError: 66162:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1381.642048] LNet: Removed LNI 192.168.204.8@tcp [ 1381.648843] LNet: Removed LNI 192.168.204.8@tcp1 [ 1381.684194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.771633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1381.802823] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1382.500822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1382.520940] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1383.347422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 1383.380665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1383.438327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1383.480916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1383.510209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1383.538374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1383.567478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1383.596417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.8@tcp [ 1383.620679] LNetError: 66435:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.8@tcp to recovery [ 1383.778450] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1386.040108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1389.920409] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1389.933814] LNet: Skipped 1 previous similar message [ 1395.125127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1395.182343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1395.242749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1402.042875] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:40:05 (1783442405) [ 1402.174554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1402.209492] LNet: 67146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1402.229170] LNetError: 67146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1402.235888] LNetError: 67146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1402.260676] LNet: Removed LNI 192.168.204.8@tcp [ 1402.289453] LNet: Removed LNI 192.168.204.8@tcp1 [ 1402.322983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1402.382622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1402.400182] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1403.397407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1403.418424] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1404.538367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 1404.628153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1404.694135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1404.764663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1404.834750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1404.922498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1404.975517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.8@tcp --health 0 [ 1405.117479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1405.409609] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.8@tcp [ 1406.440251] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1406.447657] LNet: Skipped 1 previous similar message [ 1420.869123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1421.039644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1421.082820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1421.116945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.8@tcp --health 500 [ 1421.197986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1428.088524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1428.159440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1428.187445] LNet: 67645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1428.197527] LNetError: 67645:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1428.205428] LNetError: 67645:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1428.211982] LNet: Removed LNI 192.168.204.8@tcp [ 1428.228038] LNet: Removed LNI 192.168.204.8@tcp1 [ 1428.285290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1428.350514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1428.394993] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1429.217568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1429.244691] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1430.175303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 1430.216330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1430.287967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1430.315717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1430.346638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1430.382781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1430.415586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1430.505328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.8@tcp --health 0 [ 1431.397063] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1431.407042] LNet: Skipped 3 previous similar messages [ 1431.859348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1442.160601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1442.212854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1442.263709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1450.080619] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:40:52 (1783442452) [ 1450.660684] LNet: 68692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1450.668414] LNetError: 68692:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.682352] LNetError: 68692:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1450.689706] LNet: Removed LNI 192.168.204.8@tcp [ 1450.711286] LNet: Removed LNI 192.168.204.8@tcp1 [ 1451.446693] Key type .llcrypt unregistered [ 1451.450310] Key type ._llcrypt unregistered [ 1458.432659] Key type ._llcrypt registered [ 1458.436543] Key type .llcrypt registered [ 1458.496235] LNet: 68915:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1458.496235] mask usage: [+|-] ... [ 1458.756188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1458.779433] alg: No test for adler32 (adler32-zlib) [ 1459.790123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1459.819102] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1469.387797] Lustre: DEBUG MARKER: Initial discovery [ 1469.442673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.108@tcp [ 1472.925608] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1472.981765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1473.051648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.108@tcp [ 1473.109594] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1473.165953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1473.218892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.108@tcp -r 1 -m GET -l 3 [ 1473.262949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.108@tcp -r 1 -m GET -e local_error [ 1473.299994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.108@tcp -r 1 -m PUT -l 6 [ 1474.150292] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1474.952503] Lustre: DEBUG MARKER: Force oleg408-client.virtnet to discover 192.168.204.108@tcp (in background) [ 1475.163620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.108@tcp [ 1477.211053] Lustre: DEBUG MARKER: Force oleg408-server to discover 192.168.204.8@tcp [ 1478.248183] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1480.400600] Lustre: DEBUG MARKER: Wait for 69452 [ 1481.441773] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1482.334670] Lustre: DEBUG MARKER: Finished wait on 69452 [ 1482.371807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1482.410479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1482.783881] LNet: 69904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.792266] LNetError: 69904:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1482.837811] LNet: Removed LNI 192.168.204.8@tcp [ 1483.286403] Key type .llcrypt unregistered [ 1483.289855] Key type ._llcrypt unregistered [ 1500.875747] Key type ._llcrypt registered [ 1500.887812] Key type .llcrypt registered [ 1502.109946] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:41:45 (1783442505) [ 1502.815332] Key type .llcrypt unregistered [ 1502.818507] Key type ._llcrypt unregistered [ 1510.034376] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1510.118434] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1510.123072] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1510.135669] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1512.715761] Key type ._llcrypt registered [ 1512.718899] Key type .llcrypt registered [ 1512.766867] LNet: 70951:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1512.766867] mask usage: [+|-] ... [ 1512.957594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1512.970976] alg: No test for adler32 (adler32-zlib) [ 1513.890564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.966731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1514.018827] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1514.952769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1514.976501] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1525.560051] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:42:08 (1783442528) [ 1525.683864] LNet: 71977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.689226] LNetError: 71977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1525.712211] LNet: Removed LNI 10.1.2.3@tcp [ 1525.734138] LNet: Removed LNI 192.168.204.8@tcp [ 1526.370624] Key type .llcrypt unregistered [ 1526.373408] Key type ._llcrypt unregistered [ 1533.402941] Key type ._llcrypt registered [ 1533.407646] Key type .llcrypt registered [ 1533.440963] LNet: 72235:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1533.440963] mask usage: [+|-] ... [ 1533.599109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1533.624742] alg: No test for adler32 (adler32-zlib) [ 1534.578965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.647133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1534.670530] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1535.306433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1535.341019] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1536.185179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.8@tcp [ 1537.248463] LNetError: Unexpected error -22 connecting to 192.168.204.8@tcp at host 192.168.204.8:988 [ 1542.926514] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:42:26 (1783442546) [ 1543.176344] LNet: 73117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.183889] LNetError: 73117:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1543.189723] LNet: Removed LNI 10.1.2.3@tcp [ 1544.233150] LNet: Removed LNI 192.168.204.8@tcp [ 1544.842673] Key type .llcrypt unregistered [ 1544.847362] Key type ._llcrypt unregistered [ 1550.525593] Key type ._llcrypt registered [ 1550.527928] Key type .llcrypt registered [ 1550.553964] LNet: 73349:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1550.553964] mask usage: [+|-] ... [ 1550.704418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1550.725972] alg: No test for adler32 (adler32-zlib) [ 1551.653218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.745274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1551.792746] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1552.335228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1552.356406] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 1553.199645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.8@tcp1 --nid 192.168.204.8@tcp2 [ 1557.236424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp1 192.168.204.8@tcp2 [ 1557.255699] LNetError: 73953:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.8@tcp1 to 192.168.204.8@tcp2 - Route table may be misconfigured [ 1557.280655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp2 192.168.204.8@tcp1 [ 1563.384064] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:42:46 (1783442566) [ 1563.498384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1563.525237] LNet: 74521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.532510] LNetError: 74521:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1563.538657] LNet: Removed LNI 192.168.204.8@tcp1 [ 1564.598130] LNet: Removed LNI 192.168.204.8@tcp2 [ 1564.632287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.707804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1564.746417] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1565.465585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1565.487755] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1566.449773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 1566.488493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp 192.168.204.8@tcp [ 1566.550507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp1 192.168.204.8@tcp1 [ 1566.601486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.8@tcp -d 192.168.204.8@tcp -e network_timeout [ 1566.638451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -e network_timeout [ 1566.703766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -e network_timeout [ 1566.745766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -e network_timeout [ 1566.789276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1566.813679] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1566.858044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1566.981207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1567.124895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1567.713223] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1574.850189] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:42:57 (1783442577) [ 1575.053889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.092179] LNet: 75385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.106732] LNetError: 75385:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.118821] LNetError: 75385:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1575.132146] LNet: Removed LNI 192.168.204.8@tcp [ 1576.184817] LNet: Removed LNI 192.168.204.8@tcp1 [ 1576.225712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.382793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1576.687046] LNet: 75427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.197147] Key type .llcrypt unregistered [ 1577.200210] Key type ._llcrypt unregistered [ 1591.214665] Key type ._llcrypt registered [ 1591.218579] Key type .llcrypt registered [ 1592.882485] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:43:15 (1783442595) [ 1593.163779] LNet: 76198:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1593.163779] mask usage: [+|-] ... [ 1593.303739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1593.323084] alg: No test for adler32 (adler32-zlib) [ 1594.319665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.814484] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1597.783119] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:43:20 (1783442600) [ 1597.966878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.027786] LNet: 76523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.065795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.170721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1598.207718] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1599.047395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1599.080097] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1600.231578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1600.292209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp1 [ 1600.330457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp1 [ 1607.120623] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:43:29 (1783442609) [ 1608.987821] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg408-server [ 1610.633910] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:43:33 (1783442613) [ 1612.276519] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg408-server [ 1613.920097] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:43:37 (1783442617) [ 1615.681311] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg408-server [ 1618.009117] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:43:40 (1783442620) [ 1620.992135] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg408-server [ 1623.082562] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:43:45 (1783442625) [ 1625.708386] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg408-server [ 1627.352475] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:43:50 (1783442630) [ 1629.620474] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg408-server [ 1632.164500] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:43:54 (1783442634) [ 1634.240620] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg408-server [ 1636.222129] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:43:59 (1783442639) [ 1638.249280] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg408-server [ 1640.323652] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:44:03 (1783442643) [ 1642.026916] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg408-server [ 1643.482347] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:44:06 (1783442646) [ 1643.689399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1643.714526] LNet: 80404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1643.719223] LNetError: 80404:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1643.724129] LNet: Removed LNI 192.168.204.8@tcp [ 1644.777171] LNet: Removed LNI 192.168.204.8@tcp1 [ 1644.798345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.865489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1644.914650] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1645.744853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1645.856580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1646.106890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.174250] LNet: 80552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.194169] LNetError: 80552:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1646.204851] LNetError: 80552:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1646.228159] LNet: Removed LNI 192.168.204.8@tcp [ 1646.266392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.328342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1646.351270] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1647.219598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1647.407328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.435678] LNetError: 80697:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1647.456173] LNet: Removed LNI 192.168.204.8@tcp [ 1647.479833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.546858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1647.595083] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1648.371375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1648.445807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1648.635903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.652221] LNet: 80844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1648.657511] LNet: 80844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1649.723171] LNet: Removed LNI 192.168.204.8@tcp [ 1649.773337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.909597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1649.962304] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1651.038545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1651.219707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1651.240791] LNetError: 80989:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1651.249330] LNetError: 80989:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1651.266942] LNet: Removed LNI 192.168.204.8@tcp [ 1651.310942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.379806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1651.417995] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1652.582638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1652.690412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1652.994332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.019836] LNet: 81137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.026015] LNet: 81137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1654.059808] LNet: Removed LNI 192.168.204.8@tcp [ 1654.108812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.186312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1654.211345] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1655.043712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1655.337480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.360992] LNetError: 81282:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.370188] LNetError: 81282:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1656.424799] LNet: Removed LNI 192.168.204.8@tcp [ 1656.456631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.531444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1656.550176] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1657.252482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1657.318917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1657.571961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.686498] LNet: Removed LNI 192.168.204.8@tcp [ 1658.717526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.777361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1658.830031] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1659.523119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1659.814629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.902772] LNet: Removed LNI 192.168.204.8@tcp [ 1660.920977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.001697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1661.040655] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1661.786560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1661.901676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1662.185916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.210800] LNet: 81722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.218163] LNet: 81722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1663.273140] LNet: Removed LNI 192.168.204.8@tcp [ 1663.312617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.379917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1663.432032] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1664.234331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1664.564684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.586724] LNetError: 81868:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.594288] LNetError: 81868:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1664.609361] LNet: Removed LNI 192.168.204.8@tcp [ 1664.644515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.713790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1664.744869] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1665.512257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1665.610475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1665.881640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.994935] LNet: Removed LNI 192.168.204.8@tcp [ 1667.047290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.097535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1667.128792] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1668.110540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1668.460935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1669.543989] LNet: Removed LNI 192.168.204.8@tcp [ 1669.596557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.655266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1669.688861] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1670.269724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1670.322985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1670.344837] LNet: Removed LNI 192.168.204.8@tcp [ 1670.361341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.415911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1670.451019] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1671.139575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1678.122464] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:44:41 (1783442681) [ 1678.248359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.277369] LNet: 82992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1678.288792] LNet: 82992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1678.308457] LNet: Removed LNI 192.168.204.8@tcp [ 1678.335806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.419968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1678.449255] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/180] [ 1679.612980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1679.640111] LNet: Removed LNI 192.168.204.8@tcp231 [ 1679.666708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.729872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1679.765712] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/180] [ 1679.771737] LNet: 83144:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1680.455403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1680.496065] LNet: Removed LNI 192.168.204.8@tcp231 [ 1680.526833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1680.594517] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/180] [ 1681.866724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1681.902836] LNetError: 83288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1681.911901] LNetError: 83288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1681.927848] LNet: Removed LNI 192.168.204.8@tcp231 [ 1681.971397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.050688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1682.083080] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/0] [ 1682.091585] LNet: 83296:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1682.670913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1682.707850] LNet: Removed LNI 192.168.204.8@tcp231 [ 1682.730815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1682.766391] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/0] [ 1684.303507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.334896] LNet: Removed LNI 192.168.204.8@tcp231 [ 1684.365859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.453810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1684.472875] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/60] [ 1684.477047] LNet: 83448:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1684.854461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1684.884826] LNet: Removed LNI 192.168.204.8@tcp231 [ 1684.912450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1684.953612] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/60] [ 1686.619782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.689329] LNet: Removed LNI 192.168.204.8@tcp231 [ 1686.767221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.851766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1686.898893] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/180] [ 1686.911240] LNet: 83599:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1687.423136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1687.459705] LNet: Removed LNI 192.168.204.8@tcp231 [ 1687.473957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1687.514345] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/180] [ 1695.084244] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:44:58 (1783442698) [ 1695.170645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.242278] LNet: Removed LNI 192.168.204.8@tcp231 [ 1695.275230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.335964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1695.355701] LNet: Added LNI 192.168.204.8@tcp232 [8/256/0/180] [ 1696.138143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1696.182621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1696.214966] LNet: Removed LNI 192.168.204.8@tcp232 [ 1696.237740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1696.257855] LNet: Added LNI 192.168.204.8@tcp232 [8/256/0/180] [ 1697.061620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1697.132165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1697.161584] LNet: Removed LNI 192.168.204.8@tcp232 [ 1697.185201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1697.262488] LNet: Added LNI 192.168.204.8@tcp232 [8/256/0/180] [ 1698.292844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1698.337613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1698.381264] LNet: Removed LNI 192.168.204.8@tcp232 [ 1698.409484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1698.448346] LNet: Added LNI 192.168.204.8@tcp232 [8/256/0/180] [ 1699.713828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1699.776343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1699.823241] LNet: Removed LNI 192.168.204.8@tcp232 [ 1699.842447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1708.030742] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:45:10 (1783442710) [ 1709.824057] Key type .llcrypt unregistered [ 1709.829910] Key type ._llcrypt unregistered [ 1716.876990] Key type ._llcrypt registered [ 1716.881780] Key type .llcrypt registered [ 1716.955155] LNet: 85667:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1716.955155] mask usage: [+|-] ... [ 1717.203113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1717.215330] alg: No test for adler32 (adler32-zlib) [ 1718.183402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1718.224814] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1726.339938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1736.204857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.290580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1736.318562] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1737.897248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_interrupt -r 2 [ 1737.950562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_interrupt -r 2 [ 1737.983536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_interrupt -r 2 [ 1738.036535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_interrupt -r 2 [ 1738.074860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_interrupt -r 2 [ 1738.119576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_interrupt -r 2 [ 1738.179900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_interrupt -r 2 [ 1738.224723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_interrupt -r 2 [ 1738.257960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp 192.168.204.108@tcp [ 1738.294611] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1738.306889] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1738.609537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1738.653435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1738.676495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1738.708919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1738.744460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1738.779904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1738.820313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1738.852707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1738.875821] LNet: 86101:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1738.917770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1738.960269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1739.012723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1739.060631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1739.118259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1739.184913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1739.785443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_dropped -r 2 [ 1739.825460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_dropped -r 2 [ 1739.869809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_dropped -r 2 [ 1739.901568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_dropped -r 2 [ 1739.934346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_dropped -r 2 [ 1739.966726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_dropped -r 2 [ 1740.013672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_dropped -r 2 [ 1740.057268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_dropped -r 2 [ 1740.109213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp 192.168.204.108@tcp [ 1740.141840] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1740.490887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1740.538650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1740.603595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1740.629241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1740.654599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1740.695599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1740.733887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1740.766682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1740.787130] LNet: 86222:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1740.814430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1740.866785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1740.914329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1740.958488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1741.040735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1741.109506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1741.730509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_aborted -r 2 [ 1741.759365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_aborted -r 2 [ 1741.792811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_aborted -r 2 [ 1741.833752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_aborted -r 2 [ 1741.881621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_aborted -r 2 [ 1741.947616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_aborted -r 2 [ 1741.978724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_aborted -r 2 [ 1742.018818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_aborted -r 2 [ 1742.057787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp 192.168.204.108@tcp [ 1742.092722] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1742.384937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1742.433439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1742.471940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1742.503079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1742.538876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1742.569233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1742.609746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1742.651637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1742.680662] LNet: 86342:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1742.710948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1742.755806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1742.785656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1742.843467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1742.888321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1742.923521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1743.449887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_no_route -r 2 [ 1743.508759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_no_route -r 2 [ 1743.563700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_no_route -r 2 [ 1743.607927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_no_route -r 2 [ 1743.676748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_no_route -r 2 [ 1743.720458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_no_route -r 2 [ 1743.754661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_no_route -r 2 [ 1743.803993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_no_route -r 2 [ 1743.865512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp 192.168.204.108@tcp [ 1744.166636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1744.194598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1744.229251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1744.260587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1744.295162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1744.324460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1744.371797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1744.434416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1744.483432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1744.521435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1744.571432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1744.629973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1744.668113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1744.704918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1745.368990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_timeout -r 2 [ 1745.434942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_timeout -r 2 [ 1745.499951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_timeout -r 2 [ 1745.559367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_timeout -r 2 [ 1745.599814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_timeout -r 2 [ 1745.660772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_timeout -r 2 [ 1745.701490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_timeout -r 2 [ 1745.745571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_timeout -r 2 [ 1745.781736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp 192.168.204.108@tcp [ 1745.808497] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1745.815574] LNet: Skipped 1 previous similar message [ 1746.216635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1746.253829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1746.299670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1746.333678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1746.360273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1746.388222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1746.414451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1746.443744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1746.462354] LNet: 86583:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1746.468513] LNet: 86583:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1746.484697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1746.532446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1746.585306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1746.611898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1746.642856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1746.688465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1747.370537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e remote_dropped -r 2 [ 1747.465418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e remote_dropped -r 2 [ 1747.517502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e remote_dropped -r 2 [ 1747.563905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e remote_dropped -r 2 [ 1747.607506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e remote_dropped -r 2 [ 1747.653538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e remote_dropped -r 2 [ 1747.693960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e remote_dropped -r 2 [ 1747.740423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e remote_dropped -r 2 [ 1747.791710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp 192.168.204.108@tcp [ 1747.815605] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.108@tcp [ 1748.116255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1748.157223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1748.193079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1748.253854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1748.288433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1748.346933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1748.422263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1748.488897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1748.528295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1748.554600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1748.577107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1748.603552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1748.630376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1748.658471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1754.121777] LNet: 86777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1754.130809] LNetError: 86777:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1754.146165] LNet: Removed LNI 192.168.204.8@tcp [ 1754.161306] LNet: Removed LNI 192.168.204.8@tcp1 [ 1754.611551] Key type .llcrypt unregistered [ 1754.616540] Key type ._llcrypt unregistered [ 1773.341454] Key type ._llcrypt registered [ 1773.349207] Key type .llcrypt registered [ 1774.627940] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:46:17 (1783442777) [ 1774.950454] LNet: 87584:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1774.950454] mask usage: [+|-] ... [ 1775.117824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1775.129723] alg: No test for adler32 (adler32-zlib) [ 1776.070119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1776.156874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1776.214760] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1777.226081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1777.260788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.8@tcp [ 1777.286753] LNetError: 87757:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.8@tcp to recovery [ 1777.447598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.8@tcp [ 1777.596310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1784.536464] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:46:27 (1783442787) [ 1784.662160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.688152] LNet: 88340:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1784.692595] LNetError: 88340:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1784.708584] LNet: Removed LNI 192.168.204.8@tcp [ 1784.750679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.830773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1784.878423] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1785.711714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1785.736643] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1786.887522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.8@tcp [ 1786.916413] LNetError: 88613:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.8@tcp to recovery [ 1787.872296] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1787.888440] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1788.222386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.8@tcp [ 1796.810628] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:46:39 (1783442799) [ 1798.750460] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg408-server [ 1800.888179] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:46:43 (1783442803) [ 1802.460313] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg408-server [ 1804.060392] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:46:47 (1783442807) [ 1806.053073] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg408-server [ 1808.116760] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:46:51 (1783442811) [ 1808.229518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.284421] LNet: 90242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.290458] LNetError: 90242:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.303769] LNet: Removed LNI 192.168.204.8@tcp [ 1808.317950] LNet: Removed LNI 192.168.204.8@tcp1 [ 1808.387308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.475200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1808.499493] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1809.335719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1809.391893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1809.450536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1809.541558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1809.640324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1809.681252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1816.176963] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:46:59 (1783442819) [ 1816.351128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1816.373958] LNet: 90971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1816.380787] LNetError: 90971:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1816.386410] LNetError: 90971:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1816.402355] LNet: Removed LNI 192.168.204.8@tcp [ 1816.453762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1816.569788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1816.607560] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1823.644850] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:47:06 (1783442826) [ 1823.777457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1823.804851] LNet: 91668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.820653] LNetError: 91668:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.847945] LNet: Removed LNI 192.168.204.8@tcp [ 1823.876468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.934970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1823.959893] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1831.038799] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:47:14 (1783442834) [ 1832.677630] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1834.469405] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:47:17 (1783442837) [ 1835.074748] LNet: 92710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.084317] LNetError: 92710:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.108779] LNet: Removed LNI 192.168.204.8@tcp [ 1835.887651] Key type .llcrypt unregistered [ 1835.889497] Key type ._llcrypt unregistered [ 1842.627786] Key type ._llcrypt registered [ 1842.629208] Key type .llcrypt registered [ 1842.676079] LNet: 92940:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1842.676079] mask usage: [+|-] ... [ 1842.935483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1842.976306] alg: No test for adler32 (adler32-zlib) [ 1843.967285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1843.987635] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1851.874726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1857.904141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.108@tcp [ 1858.254995] LNet: 93129:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.268939] LNetError: 93129:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1858.282780] LNet: Removed LNI 192.168.204.8@tcp [ 1858.714594] Key type .llcrypt unregistered [ 1858.720665] Key type ._llcrypt unregistered [ 1870.423747] Key type ._llcrypt registered [ 1870.426793] Key type .llcrypt registered [ 1871.819557] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1873.490414] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1875.004518] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:47:58 (1783442878) [ 1876.699754] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1878.259983] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:48:01 (1783442881) [ 1879.801895] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg408-server [ 1881.592525] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:48:04 (1783442884) [ 1883.657449] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg408-server [ 1885.369689] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:48:08 (1783442888) [ 1886.328664] LNet: 95180:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1886.328664] mask usage: [+|-] ... [ 1886.934471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1886.960455] alg: No test for adler32 (adler32-zlib) [ 1888.053587] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 1888.391473] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1890.120186] Key type lgssc registered [ 1891.567459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1915.904206] Key type lgssc unregistered [ 1916.155559] LNet: 96304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.162717] LNetError: 96304:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.180191] LNet: Removed LNI 192.168.204.8@tcp [ 1917.090543] Key type .llcrypt unregistered [ 1917.092453] Key type ._llcrypt unregistered [ 1932.493948] Key type ._llcrypt registered [ 1932.501932] Key type .llcrypt registered [ 1933.930905] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:48:56 (1783442936) [ 1934.204121] LNet: 97097:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1934.204121] mask usage: [+|-] ... [ 1934.406652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1934.417960] alg: No test for adler32 (adler32-zlib) [ 1935.436338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1935.545613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1935.634270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1935.717864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1935.802608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1943.321800] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:49:06 (1783442946) [ 1943.631839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.655716] LNet: 97718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.681746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.735811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1943.757644] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1944.642564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1944.667634] LNet: 97862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.678373] LNetError: 97862:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.692341] LNet: Removed LNI 192.168.204.8@tcp [ 1944.717550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1944.813963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1944.874894] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1945.344194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1945.370421] LNetError: 97890:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.409039] LNet: Removed LNI 192.168.204.8@tcp [ 1945.437753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1945.527223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1945.571713] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1953.190991] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:49:15 (1783442955) [ 1953.313912] LNet: 98480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1953.322396] LNet: 98480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1953.338763] LNetError: 98480:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1953.361030] LNet: Removed LNI 192.168.204.8@tcp [ 1954.046216] Key type .llcrypt unregistered [ 1954.052404] Key type ._llcrypt unregistered [ 1960.819811] Key type ._llcrypt registered [ 1960.822580] Key type .llcrypt registered [ 1960.877304] LNet: 98712:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1960.877304] mask usage: [+|-] ... [ 1961.067449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1961.085550] alg: No test for adler32 (adler32-zlib) [ 1962.068503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.141904] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1962.149505] LNet: *** cfs_fail_loc=e003, val=100*** [ 1962.881638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1962.897502] LNet: *** cfs_fail_loc=e003, val=100*** [ 1962.900258] LNet: Skipped 2 previous similar messages [ 1962.904033] Ping reply received with mapping: [ 1962.905928] found metadata reply NI[192.168.204.8@tcp] [ 1962.908962] 08 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1962.912177] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1962.914990] 5a 5a 5a 5a 5a 5a 5a 5a [ 1963.056512] LNet: 98878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.068200] LNetError: 98878:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.086471] LNet: Removed LNI 192.168.204.8@tcp [ 1963.664603] Key type .llcrypt unregistered [ 1963.667695] Key type ._llcrypt unregistered [ 1974.076563] Key type ._llcrypt registered [ 1974.079839] Key type .llcrypt registered [ 1975.176389] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:49:38 (1783442978) [ 1975.677993] Key type .llcrypt unregistered [ 1975.681592] Key type ._llcrypt unregistered [ 1981.166755] Key type ._llcrypt registered [ 1981.169674] Key type .llcrypt registered [ 1981.195571] LNet: 99874:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1981.195571] mask usage: [+|-] ... [ 1981.322433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1981.336299] alg: No test for adler32 (adler32-zlib) [ 1982.488214] LNetError: 99924:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1982.782634] Key type .llcrypt unregistered [ 1982.790254] Key type ._llcrypt unregistered [ 1987.398791] Key type ._llcrypt registered [ 1987.411792] Key type .llcrypt registered [ 1988.731775] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:49:51 (1783442991) [ 1989.167123] Key type .llcrypt unregistered [ 1989.169850] Key type ._llcrypt unregistered [ 1994.560183] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1994.566828] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1997.058384] Key type ._llcrypt registered [ 1997.061497] Key type .llcrypt registered [ 1997.093501] LNet: 100791:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1997.093501] mask usage: [+|-] ... [ 1997.233468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1997.245356] alg: No test for adler32 (adler32-zlib) [ 1998.212225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1998.261969] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2007.094698] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:50:10 (1783443010) [ 2007.248922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2007.275723] LNet: 101704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.281647] LNetError: 101704:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2007.294421] LNet: Removed LNI 10.1.2.3@tcp [ 2007.313558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2007.909356] Key type .llcrypt unregistered [ 2007.914419] Key type ._llcrypt unregistered [ 2013.293134] Key type ._llcrypt registered [ 2013.297267] Key type .llcrypt registered [ 2013.328905] LNet: 101935:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2013.328905] mask usage: [+|-] ... [ 2013.495143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2013.521600] alg: No test for adler32 (adler32-zlib) [ 2014.428962] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2014.494907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 2014.699957] LNet: 101993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2014.707197] LNetError: 101993:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2015.782568] LNet: Removed LNI 192.168.204.8@tcp [ 2016.481686] Key type .llcrypt unregistered [ 2016.483652] Key type ._llcrypt unregistered [ 2025.505069] Key type ._llcrypt registered [ 2025.512975] Key type .llcrypt registered [ 2026.695425] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:50:29 (1783443029) [ 2027.289603] Key type .llcrypt unregistered [ 2027.292479] Key type ._llcrypt unregistered [ 2034.328292] Key type ._llcrypt registered [ 2034.330462] Key type .llcrypt registered [ 2034.384137] LNet: 102977:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2034.384137] mask usage: [+|-] ... [ 2034.484824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2034.498428] alg: No test for adler32 (adler32-zlib) [ 2043.281365] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:50:46 (1783443046) [ 2043.510776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2043.604289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2043.716409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2043.801698] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2044.755679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2044.783799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2044.825536] LNet: Removed LNI 192.168.204.8@tcp [ 2045.146441] LNet: 103736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.749662] Key type .llcrypt unregistered [ 2045.757056] Key type ._llcrypt unregistered [ 2058.204752] Key type ._llcrypt registered [ 2058.217489] Key type .llcrypt registered [ 2059.599357] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:51:02 (1783443062) [ 2060.164820] LNet: 104502:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2060.164820] mask usage: [+|-] ... [ 2060.441810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2060.475325] alg: No test for adler32 (adler32-zlib) [ 2061.464867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.542445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2061.575378] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2062.235945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 2062.308743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.8@tcp [ 2062.328710] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2062.332406] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2062.335294] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2062, 0, 0 [ 2062.339773] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2062.341601] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2062.346629] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2062.356309] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2062, 0, 0 [ 2062.368315] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2062.372799] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2062.378734] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2062.381326] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2062, 0, 2112 [ 2062.394115] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2062.400963] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2062.408635] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2062.413150] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2062, 0, 2112 [ 2062.420318] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2069.634740] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:51:12 (1783443072) [ 2070.154899] LNet: 105276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2070.172879] LNetError: 105276:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2070.226855] LNet: Removed LNI 192.168.204.8@tcp [ 2070.925285] Key type .llcrypt unregistered [ 2070.933707] Key type ._llcrypt unregistered [ 2077.880474] Key type ._llcrypt registered [ 2077.883351] Key type .llcrypt registered [ 2077.931506] LNet: 105499:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2077.931506] mask usage: [+|-] ... [ 2078.144709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2078.165738] alg: No test for adler32 (adler32-zlib) [ 2079.081427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2079.105293] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2086.970434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 2095.357312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2095.410538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2095.441352] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 2096.249796] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2096.360144] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2096.371124] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2096.388818] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2098.803552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2098.832583] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2110.053193] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:51:52 (1783443112) [ 2110.173108] LNet: 106775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.195504] LNetError: 106775:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2110.205557] LNet: Removed LNI 192.168.204.8@tcp [ 2111.264326] LNet: Removed LNI 192.168.204.8@tcp1 [ 2111.268846] LNetError: 106775:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2111.275584] LNetError: 106775:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2111.295901] LNet: Removed LNI 10.1.2.3@tcp99 [ 2112.100748] Key type .llcrypt unregistered [ 2112.102837] Key type ._llcrypt unregistered [ 2118.665160] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2118.672514] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2120.159355] Key type ._llcrypt registered [ 2120.162707] Key type .llcrypt registered [ 2120.198305] LNet: 107040:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2120.198305] mask usage: [+|-] ... [ 2120.342756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2120.356675] alg: No test for adler32 (adler32-zlib) [ 2121.284288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.357187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2121.382555] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2122.049795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2122.070119] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2124.160133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2124.197541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2124.239722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2124.262939] LNetError: 107518:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2124.311845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2124.377849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2124.807165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2124.896672] LNet: 107538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2124.904873] LNetError: 107538:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2124.917373] LNet: Removed LNI 10.1.2.3@tcp [ 2125.990813] LNet: Removed LNI 192.168.204.8@tcp [ 2126.801930] Key type .llcrypt unregistered [ 2126.805455] Key type ._llcrypt unregistered [ 2137.472211] Key type ._llcrypt registered [ 2137.474915] Key type .llcrypt registered [ 2138.807811] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:52:21 (1783443141) [ 2139.091104] LNet: 108337:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2139.091104] mask usage: [+|-] ... [ 2139.231837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2139.242733] alg: No test for adler32 (adler32-zlib) [ 2140.156456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.218990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2140.246243] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2140.903538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 2140.964844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg408-client@tcp [ 2147.447534] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:52:30 (1783443150) [ 2147.833104] LNet: 109073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2147.845868] LNetError: 109073:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2147.875961] LNet: Removed LNI 192.168.204.8@tcp [ 2148.591418] Key type .llcrypt unregistered [ 2148.593253] Key type ._llcrypt unregistered [ 2154.903091] Key type ._llcrypt registered [ 2154.910159] Key type .llcrypt registered [ 2155.088543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2155.133781] alg: No test for adler32 (adler32-zlib) [ 2156.018418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2156.037552] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2156.063364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2161.630383] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:52:44 (1783443164) [ 2163.177886] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2164.836220] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:52:47 (1783443167) [ 2166.204910] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2168.104355] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:52:51 (1783443171) [ 2168.223720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2168.252577] LNet: 110486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.265046] LNetError: 110486:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.293966] LNet: Removed LNI 192.168.204.8@tcp [ 2168.319373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2168.394875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2168.425551] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2169.059024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2169.133949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2174.829212] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:52:57 (1783443177) [ 2174.926811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2174.951992] LNet: 111195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2174.957450] LNetError: 111195:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2174.977832] LNet: Removed LNI 192.168.204.8@tcp [ 2174.994340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2175.049897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2175.084523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2175.754556] Key type .llcrypt unregistered [ 2175.758347] Key type ._llcrypt unregistered [ 2186.712792] Key type ._llcrypt registered [ 2186.714391] Key type .llcrypt registered [ 2187.780192] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:53:10 (1783443190) [ 2188.084694] LNet: 112010:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2188.084694] mask usage: [+|-] ... [ 2188.212488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2188.230559] alg: No test for adler32 (adler32-zlib) [ 2189.124803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.156858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2189.174789] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2189.698878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2189.739165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2194.644775] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:53:18 (1783443198) [ 2194.779886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2194.818341] LNet: 112756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2194.834326] LNetError: 112756:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2194.863718] LNet: Removed LNI 192.168.204.8@tcp [ 2194.888709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2194.951027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2194.986593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2199.762420] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:53:23 (1783443203) [ 2199.929250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2199.956981] LNet: 113337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2199.981860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2200.039654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2200.427981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2200.465595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2200.546851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2200.880183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2200.904426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2200.974709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2201.002456] LNet: 113371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2201.030845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2201.089381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2201.347853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2201.373320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2201.419199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2201.799719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2201.834323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2207.262747] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:53:30 (1783443210) [ 2207.445557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2207.476113] LNet: 113971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2207.499037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2207.543528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2207.565678] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2208.262940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2208.302800] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2209.226213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2209.290844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2209.521235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2209.546803] LNetError: 114250:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2210.592167] LNet: Removed LNI 192.168.204.8@tcp [ 2210.595099] LNetError: 114250:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2210.613936] LNet: Removed LNI 192.168.204.8@tcp2 [ 2210.634351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.734456] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2210.746504] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2210.754581] LNet: 114256:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2211.387294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2211.570814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2211.598118] LNet: 114294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2211.607444] LNet: 114294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2212.640831] LNet: Removed LNI 192.168.204.8@tcp [ 2212.643296] LNetError: 114294:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2212.654927] LNetError: 114294:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2212.674723] LNet: Removed LNI 192.168.204.8@tcp2 [ 2212.713333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.790835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2212.815677] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2213.373394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2213.390978] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2214.294901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2214.347913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2214.408618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2214.590251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2215.648226] LNet: Removed LNI 192.168.204.8@tcp [ 2216.679537] LNet: Removed LNI 192.168.204.8@tcp2 [ 2216.733837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.845917] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2216.854394] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2216.866273] LNet: 114584:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2217.453360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2217.677297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2217.693685] LNetError: 114621:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2217.699719] LNetError: 114621:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2217.705755] LNet: Removed LNI 192.168.204.8@tcp [ 2218.726842] LNet: Removed LNI 192.168.204.8@tcp2 [ 2218.753395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2218.813069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2218.845525] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2219.420227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2219.446833] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2220.253145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2220.324331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2220.647953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2220.714984] LNet: 114901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.723302] LNet: 114901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2221.792973] LNet: Removed LNI 192.168.204.8@tcp [ 2221.812361] LNet: Removed LNI 192.168.204.8@tcp2 [ 2221.837659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2221.926841] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2221.938340] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2221.945320] LNet: 114907:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2222.633567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2222.828658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2223.904199] LNet: Removed LNI 192.168.204.8@tcp [ 2223.920567] LNet: Removed LNI 192.168.204.8@tcp2 [ 2223.954223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2224.027784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2224.061542] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2224.716209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2224.730181] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2225.534664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2225.566224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2225.610607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2225.814426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2225.834086] LNetError: 115229:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2225.839407] LNetError: 115229:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2225.846217] LNet: Removed LNI 192.168.204.8@tcp [ 2226.925521] LNet: Removed LNI 192.168.204.8@tcp2 [ 2226.980451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.066694] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2227.076941] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2227.085623] LNet: 115235:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2227.754953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2228.083586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2228.129358] LNet: Removed LNI 192.168.204.8@tcp [ 2229.165235] LNet: Removed LNI 192.168.204.8@tcp2 [ 2229.196258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2229.233575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2229.253924] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2229.813902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2229.833373] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2231.004257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2231.061775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2231.131750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2231.562356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2232.608218] LNet: Removed LNI 192.168.204.8@tcp [ 2232.623179] LNet: Removed LNI 192.168.204.8@tcp2 [ 2232.671214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2232.768704] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2232.777108] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2232.786948] LNet: 115575:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2233.437414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2233.832803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.921588] LNet: Removed LNI 192.168.204.8@tcp [ 2234.934620] LNet: Removed LNI 192.168.204.8@tcp2 [ 2234.982186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.065438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2235.108025] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2235.703419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2235.732902] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2236.618172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2236.691623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2236.733724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2236.798971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2237.106975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2237.122520] LNet: 115928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2237.129349] LNet: 115928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2238.176585] LNet: Removed LNI 192.168.204.8@tcp [ 2239.209624] LNet: Removed LNI 192.168.204.8@tcp2 [ 2239.248779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.328977] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2239.333456] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2239.964185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2246.028900] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2247.297545] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:54:10 (1783443250) [ 2247.443888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2247.465792] LNetError: 116693:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2247.478856] LNetError: 116693:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2247.490036] LNet: Removed LNI 192.168.204.8@tcp [ 2248.551835] LNet: Removed LNI 192.168.204.8@tcp2 [ 2248.588386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.882294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2248.906216] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2249.543037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2249.563648] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2250.360293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2250.383382] LNet: Removed LNI 10.1.2.3@tcp [ 2250.394221] LNet: Removed LNI 192.168.204.8@tcp [ 2250.409463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2250.465317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2250.486470] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2250.491293] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2250.887456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2250.903967] LNet: Removed LNI 192.168.204.8@tcp [ 2250.919785] LNet: Removed LNI 10.1.2.3@tcp [ 2256.102221] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:54:19 (1783443259) [ 2256.177824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2256.229666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.312235] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2256.396911] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2256.404321] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2256.407918] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2258.761221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2258.801950] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2259.576523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2259.613519] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2261.635590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2261.662623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2261.711889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.8@tcp -m PUT -e local_timeout -r 1 [ 2261.760454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2261.811880] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2261.817480] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2262.001452] LNetError: 117635:0:(lib-move.c:4714:lnet_parse()) 192.168.204.8@tcp, src 192.168.204.8@tcp: Dropping PUT (error -108 looking up sender) [ 2263.013794] LNet: Removed LNI 10.1.2.3@tcp [ 2264.040214] LNet: Removed LNI 192.168.204.8@tcp [ 2264.339536] Key type .llcrypt unregistered [ 2264.346615] Key type ._llcrypt unregistered [ 2273.787832] Key type ._llcrypt registered [ 2273.793703] Key type .llcrypt registered [ 2274.885648] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:54:38 (1783443278) [ 2275.055314] LNet: 118862:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2275.055314] mask usage: [+|-] ... [ 2275.186391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2275.204698] alg: No test for adler32 (adler32-zlib) [ 2276.093275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2276.388284] LNet: 118945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.885306] Key type .llcrypt unregistered [ 2276.887509] Key type ._llcrypt unregistered [ 2282.695449] Key type ._llcrypt registered [ 2282.697808] Key type .llcrypt registered [ 2282.720819] LNet: 119174:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2282.720819] mask usage: [+|-] ... [ 2282.886228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2282.906912] alg: No test for adler32 (adler32-zlib) [ 2283.791983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2283.821075] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2290.760979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 2290.967341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2291.501350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.108@tcp [ 2291.751055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2291.774909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2291.803795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2291.831441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2291.858766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2291.899234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2291.924170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2291.947384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2291.959424] LNet: 119394:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2291.974430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2291.999356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2292.015560] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.108@tcp [ 2292.035782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2292.072517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2292.120879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2292.149578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2292.820102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2293.034155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2293.064248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2293.104922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2293.179155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2293.213657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2293.263360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2293.307687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2293.338737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2293.356508] LNet: 119511:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2293.387413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2293.422165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2293.454824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2293.485825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2293.523146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2293.556597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2294.154156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.108@tcp [ 2294.370330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2294.402442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2294.433393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2294.463211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2294.511837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2294.545832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2294.585102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2294.628434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2294.663772] LNet: 119629:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2294.690603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2294.736668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2294.769605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2294.804423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2294.834591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2294.877283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2295.465589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2295.669630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2295.706937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2295.740495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2295.775814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2295.803291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2295.841520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2295.895991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2295.924559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2295.959266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2296.023197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2296.055617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2296.091761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2296.125934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2296.149625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2296.469173] LNet: 119792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.479290] LNetError: 119792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2296.493474] LNet: Removed LNI 192.168.204.8@tcp [ 2296.805589] Key type .llcrypt unregistered [ 2296.809140] Key type ._llcrypt unregistered [ 2310.113905] Key type ._llcrypt registered [ 2310.116112] Key type .llcrypt registered [ 2311.225446] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:55:14 (1783443314) [ 2311.578310] LNet: 120597:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2311.578310] mask usage: [+|-] ... [ 2311.747745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2311.770985] alg: No test for adler32 (adler32-zlib) [ 2312.711336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2313.053791] LNet: 120678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2313.658297] Key type .llcrypt unregistered [ 2313.660431] Key type ._llcrypt unregistered [ 2319.412558] Key type ._llcrypt registered [ 2319.416053] Key type .llcrypt registered [ 2319.459548] LNet: 120907:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2319.459548] mask usage: [+|-] ... [ 2319.697386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2319.712316] alg: No test for adler32 (adler32-zlib) [ 2320.565840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2320.591625] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2327.367522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 2334.901767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.934582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2334.960181] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 2335.689693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2336.164648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.108@tcp [ 2336.333681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2336.357923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2336.385782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2336.415178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2336.444383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2336.471640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2336.510844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2336.539972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2336.554331] LNet: 121330:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2336.568599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2336.596509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2336.626252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2336.660210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2336.688356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2336.712640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2337.272746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2337.484059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2337.507326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2337.534649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2337.573739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2337.607386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2337.631816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2337.657159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2337.688125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2337.702533] LNet: 121447:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2337.716774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2337.742474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2337.768689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2337.791849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2337.815142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2337.839856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2338.422496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.108@tcp [ 2338.640943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2338.687871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2338.721515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2338.751620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2338.780539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2338.813575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2338.851840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2338.885845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2338.907297] LNet: 121565:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2338.924566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2338.950579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2338.978596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2339.007404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2339.033192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2339.040365] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.108@tcp [ 2339.062240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2339.640680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2339.850515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2339.877339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2339.911848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2339.952764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2339.983689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2340.016512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2340.048961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2340.082449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2340.111827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2340.145830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2340.188087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2340.218604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2340.258053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2340.288159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2344.545890] LNet: 121763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.552404] LNetError: 121763:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.559546] LNet: Removed LNI 192.168.204.8@tcp [ 2344.569664] LNet: Removed LNI 192.168.204.8@tcp1 [ 2344.928630] Key type .llcrypt unregistered [ 2344.935402] Key type ._llcrypt unregistered [ 2357.045480] Key type ._llcrypt registered [ 2357.047170] Key type .llcrypt registered [ 2357.905862] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:56:01 (1783443361) [ 2363.069201] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:56:06 (1783443366) [ 2367.985800] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:56:11 (1783443371) [ 2372.351342] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:56:15 (1783443375) [ 2376.764762] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:56:20 (1783443380) [ 2381.524314] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:56:25 (1783443385) [ 2385.790962] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:56:29 (1783443389) [ 2390.228822] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:56:33 (1783443393) [ 2395.006971] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:56:38 (1783443398) [ 2399.460824] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:56:42 (1783443402) [ 2403.720809] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:56:47 (1783443407) [ 2407.951856] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:56:51 (1783443411) [ 2412.292825] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:56:55 (1783443415) [ 2417.074231] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:57:00 (1783443420) [ 2421.947560] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:57:05 (1783443425) [ 2426.271366] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:57:09 (1783443429) [ 2431.082988] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:57:14 (1783443434) [ 2436.397056] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:57:19 (1783443439) [ 2436.528534] LNet: 132294:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2436.528534] mask usage: [+|-] ... [ 2436.653494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2436.669876] alg: No test for adler32 (adler32-zlib) [ 2437.527907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2437.611188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2437.625856] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2438.246539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2438.263501] LNet: 132466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2438.269511] LNetError: 132466:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2438.280710] LNet: Removed LNI 192.168.204.8@tcp [ 2438.291954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2438.324127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2438.346554] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2442.428150] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:57:26 (1783443446) [ 2447.000616] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:57:30 (1783443450) [ 2451.537907] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:57:35 (1783443455) [ 2456.093690] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:57:39 (1783443459) [ 2460.610190] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:57:44 (1783443464) [ 2465.552509] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:57:48 (1783443468) [ 2469.821922] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:57:53 (1783443473) [ 2474.323889] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:57:57 (1783443477) [ 2479.178190] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:58:02 (1783443482) [ 2484.375870] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:58:07 (1783443487) [ 2488.885749] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:58:12 (1783443492) [ 2493.399407] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:58:16 (1783443496) [ 2497.779119] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2213 sec ========== 12:58:21 (1783443501) [ 2498.044234] LNet: 139927:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2498.055204] LNetError: 139927:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2498.069625] LNet: Removed LNI 192.168.204.8@tcp [ 2498.571104] Key type .llcrypt unregistered [ 2498.573584] Key type ._llcrypt unregistered [ 2506.762720] Key type ._llcrypt registered [ 2506.765540] Key type .llcrypt registered [ 2506.813632] LNet: 140403:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2506.813632] mask usage: [+|-] ... [ 2506.942232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2506.960476] alg: No test for adler32 (adler32-zlib) [ 2507.964811] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 2508.143189] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2509.800227] Key type lgssc registered [ 2510.635299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2554.437257] Lustre: Mounted lustre-client [ 2554.569589] LNet: 141870:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2554.569589] mask usage: [+|-] ... [ 2558.302176] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2563.522962] Lustre: Unmounted lustre-client [ 2595.013448] Key type lgssc unregistered [ 2595.189601] LNet: 142851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2595.196358] LNetError: 142851:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2595.212799] LNet: Removed LNI 192.168.204.8@tcp [ 2595.638206] Key type .llcrypt unregistered [ 2595.640422] Key type ._llcrypt unregistered