[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 460064928 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003146] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008490] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011138] LSM: Security Framework initializing [ 0.013039] Yama: becoming mindful. [ 0.014039] SELinux: Initializing. [ 0.015073] *** VALIDATE selinux *** [ 0.023142] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027564] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030114] *** VALIDATE tmpfs *** [ 0.031468] *** VALIDATE proc *** [ 0.033202] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.036143] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039031] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043563] debug: unmapping init [mem 0xffffffff8ec59000-0xffffffff8ec60fff] [ 0.045178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046713] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047024] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052012] ... fixed-purpose events: 3 [ 0.053010] ... event mask: 000000070000000f [ 0.054290] rcu: Hierarchical SRCU implementation. [ 0.056442] smp: Bringing up secondary CPUs ... [ 0.057617] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.061398] smp: Brought up 1 node, 4 CPUs [ 0.063012] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142359] node 0 deferred pages initialised in 76ms [ 0.145529] devtmpfs: initialized [ 0.147277] x86/mm: Memory block size: 128MB [ 0.151100] gcov: version magic: 0x41383552 [ 0.155303] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159119] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161323] pinctrl core: initialized pinctrl subsystem [ 0.163179] [ 0.163799] ************************************************************* [ 0.166010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168010] ** ** [ 0.171012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.173010] ** ** [ 0.175012] ** This means that this kernel is built to expose internal ** [ 0.177009] ** IOMMU data structures, which may compromise security on ** [ 0.179009] ** your system. ** [ 0.181009] ** ** [ 0.183011] ** If you see this message and you are not debugging the ** [ 0.185010] ** kernel, report this immediately to your vendor! ** [ 0.187013] ** ** [ 0.189010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192013] ************************************************************* [ 0.194641] NET: Registered protocol family 16 [ 0.196427] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.198117] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.201053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.204431] cpuidle: using governor menu [ 0.207367] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.209520] PCI: Using configuration type 1 for base access [ 0.211130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.222357] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.224020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227074] cryptd: max_cpu_qlen set to 1000 [ 0.230276] ACPI: Added _OSI(Module Device) [ 0.232020] ACPI: Added _OSI(Processor Device) [ 0.234012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.241585] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248502] ACPI: Interpreter enabled [ 0.250063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.252014] ACPI: Using IOAPIC for interrupt routing [ 0.254091] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257368] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.270000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.272039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.275020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.278085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284455] acpiphp: Slot [2] registered [ 0.286112] acpiphp: Slot [5] registered [ 0.288102] acpiphp: Slot [6] registered [ 0.289088] acpiphp: Slot [3] registered [ 0.290037] acpiphp: Slot [4] registered [ 0.291115] acpiphp: Slot [7] registered [ 0.292082] acpiphp: Slot [8] registered [ 0.293157] acpiphp: Slot [9] registered [ 0.295077] acpiphp: Slot [10] registered [ 0.296088] acpiphp: Slot [11] registered [ 0.298134] acpiphp: Slot [12] registered [ 0.299087] acpiphp: Slot [13] registered [ 0.301079] acpiphp: Slot [14] registered [ 0.302151] acpiphp: Slot [15] registered [ 0.303085] acpiphp: Slot [16] registered [ 0.304037] acpiphp: Slot [17] registered [ 0.306111] acpiphp: Slot [18] registered [ 0.307204] acpiphp: Slot [19] registered [ 0.309095] acpiphp: Slot [20] registered [ 0.310144] acpiphp: Slot [21] registered [ 0.311098] acpiphp: Slot [22] registered [ 0.312191] acpiphp: Slot [23] registered [ 0.314080] acpiphp: Slot [24] registered [ 0.315131] acpiphp: Slot [25] registered [ 0.316063] acpiphp: Slot [26] registered [ 0.317065] acpiphp: Slot [27] registered [ 0.319230] acpiphp: Slot [28] registered [ 0.321174] acpiphp: Slot [29] registered [ 0.323106] acpiphp: Slot [30] registered [ 0.325118] acpiphp: Slot [31] registered [ 0.327121] PCI host bridge to bus 0000:00 [ 0.328016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.334034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.337027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.339030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.342086] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345225] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.348085] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.351520] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.360013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.365008] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.366017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.367018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.370016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.372873] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.375770] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.379041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.381821] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.385960] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.397998] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.402816] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.409275] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.419017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.426017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.449019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.460608] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.473014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.479016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.496017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.510768] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.512358] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.514326] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.516242] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.518233] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.524565] iommu: Default domain type: Passthrough [ 0.525388] SCSI subsystem initialized [ 0.526091] ACPI: bus type USB registered [ 0.527083] usbcore: registered new interface driver usbfs [ 0.529068] usbcore: registered new interface driver hub [ 0.530000] usbcore: registered new device driver usb [ 0.530000] pps_core: LinuxPPS API ver. 1 registered [ 0.531009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.533482] PTP clock support registered [ 0.535125] EDAC MC: Ver: 3.0.0 [ 0.536168] PCI: Using ACPI for IRQ routing [ 0.537846] NetLabel: Initializing [ 0.539010] NetLabel: domain hash size = 128 [ 0.540006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.541063] NetLabel: unlabeled traffic allowed by default [ 0.543127] vgaarb: loaded [ 0.545185] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.546006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.551496] clocksource: Switched to clocksource kvm-clock [ 0.648393] VFS: Disk quotas dquot_6.6.0 [ 0.649992] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.652479] *** VALIDATE ramfs *** [ 0.653798] *** VALIDATE hugetlbfs *** [ 0.655416] pnp: PnP ACPI init [ 0.658447] pnp: PnP ACPI: found 6 devices [ 0.673839] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.676023] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.677464] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.679215] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.681786] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.684437] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.687523] NET: Registered protocol family 2 [ 0.690024] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.696757] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.700759] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.708067] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.711764] TCP: Hash tables configured (established 65536 bind 65536) [ 0.714720] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.719074] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.722114] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.725500] NET: Registered protocol family 1 [ 0.728532] RPC: Registered named UNIX socket transport module. [ 0.730591] RPC: Registered udp transport module. [ 0.732092] RPC: Registered tcp transport module. [ 0.733618] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.735869] NET: Registered protocol family 44 [ 0.737736] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.740120] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.742421] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.744985] PCI: CLS 0 bytes, default 64 [ 0.746939] Unpacking initramfs... [ 2.101525] debug: unmapping init [mem 0xffff9ef17cc64000-0xffff9ef17ffcffff] [ 2.107156] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.109557] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.113236] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.593605] Initialise system trusted keyrings [ 2.594925] Key type blacklist registered [ 2.596375] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.604081] zbud: loaded [ 2.607418] *** VALIDATE nfs *** [ 2.608347] *** VALIDATE nfs4 *** [ 2.609455] pstore: using deflate compression [ 2.612357] Platform Keyring initialized [ 2.726650] NET: Registered protocol family 38 [ 2.728178] Key type asymmetric registered [ 2.729530] Asymmetric key parser 'x509' registered [ 2.731141] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.734351] io scheduler mq-deadline registered [ 2.735866] io scheduler kyber registered [ 2.737354] io scheduler bfq registered [ 2.740142] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.742976] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.745526] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.748141] ACPI: Power Button [PWRF] [ 2.753352] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.759603] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.769040] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.796271] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.824052] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.828173] Non-volatile memory driver v1.3 [ 2.829582] Linux agpgart interface v0.103 [ 2.856394] virtio_blk virtio1: [vda] 144824 512-byte logical blocks (74.1 MB/70.7 MiB) [ 2.858133] vda: detected capacity change from 0 to 74149888 [ 2.872409] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.875436] vdb: detected capacity change from 0 to 1073741824 [ 2.886812] libphy: Fixed MDIO Bus: probed [ 2.911937] usbcore: registered new interface driver usbserial_generic [ 2.914298] usbserial: USB Serial support registered for generic [ 2.916621] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.921792] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.923659] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.927131] mousedev: PS/2 mouse device common for all mice [ 2.930335] rtc_cmos 00:05: RTC can wake from S4 [ 2.935534] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.938787] rtc_cmos 00:05: registered as rtc0 [ 2.941872] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.946088] intel_pstate: CPU model not supported [ 2.947072] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.951079] hid: raw HID events driver (C) Jiri Kosina [ 2.956819] usbcore: registered new interface driver usbhid [ 2.958484] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.958899] usbhid: USB HID core driver [ 2.963109] drop_monitor: Initializing network drop monitor service [ 2.966093] Initializing XFRM netlink socket [ 2.968403] NET: Registered protocol family 10 [ 2.971739] Segment Routing with IPv6 [ 2.972866] NET: Registered protocol family 17 [ 2.974602] mpls_gso: MPLS GSO support [ 2.980756] RAS: Correctable Errors collector initialized. [ 2.983629] AVX version of gcm_enc/dec engaged. [ 2.985228] AES CTR mode by8 optimization enabled [ 3.052367] sched_clock: Marking stable (3052344687, 0)->(3972512984, -920168297) [ 3.055223] registered taskstats version 1 [ 3.056386] Loading compiled-in X.509 certificates [ 3.057793] zswap: loaded using pool lzo/zbud [ 3.075448] Key type big_key registered [ 3.085547] Key type encrypted registered [ 3.086611] ima: No TPM chip found, activating TPM-bypass! [ 3.087974] ima: Allocated hash algorithm: sha1 [ 3.090354] ima: No architecture policies found [ 3.091522] evm: Initialising EVM extended attributes: [ 3.092602] evm: security.selinux [ 3.093515] evm: security.ima [ 3.094208] evm: security.capability [ 3.094874] evm: HMAC attrs: 0x1 [ 3.096545] rtc_cmos 00:05: setting system clock to 2026-06-28 00:55:38 UTC (1782608138) [ 3.100388] debug: unmapping init [mem 0xffffffff8fc03000-0xffffffff8fdfffff] [ 3.102254] debug: unmapping init [mem 0xffffffff8e982000-0xffffffff8ec58fff] [ 3.110103] Write protecting the kernel read-only data: 28672k [ 3.112101] debug: unmapping init [mem 0xffffffff8d003000-0xffffffff8d1fffff] [ 3.113802] debug: unmapping init [mem 0xffffffff8d914000-0xffffffff8d9fffff] [ 3.136041] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.140924] systemd[1]: Detected virtualization kvm. [ 3.142161] systemd[1]: Detected architecture x86-64. [ 3.143386] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.170081] systemd[1]: No hostname configured. [ 3.171638] systemd[1]: Set hostname to . [ 3.173228] random: systemd: uninitialized urandom read (16 bytes read) [ 3.175983] systemd[1]: Initializing machine ID from random generator. [ 3.216285] random: ln: uninitialized urandom read (6 bytes read) [ 3.311244] random: systemd: uninitialized urandom read (16 bytes read) [ 3.313192] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.320196] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 3.329078] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.057130] device-mapper: uevent: version 1.0.3 [ 4.059411] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.730770] virtio_net virtio0 ens2: renamed from eth0 [ 4.898907] scsi host0: ata_piix [ 4.960212] scsi host1: ata_piix [ 4.961823] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.963897] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.076283] dracut-initqueue[601]: RTNETLINK answers: File exists [ 9.599338] random: crng init done [ 9.600774] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.009465] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.273799] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.565863] SELinux: Disabled at runtime. [ 11.627737] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.636554] systemd[1]: Detected virtualization kvm. [ 11.638597] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.178473] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.181954] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.185868] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.192461] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.196208] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.203453] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.215608] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ 12.236946] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.788515] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.162151] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.236271] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.368657] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.384204] EDAC sbridge: Ver: 1.1.2 [ 14.794331] Key type dns_resolver registered [ 15.092338] NFS: Registering the id_resolver key type [ 15.093986] Key type id_resolver registered [ 15.095140] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg143-client login: [ 68.663033] libcfs: loading out-of-tree module taints kernel. [ 68.973443] Key type ._llcrypt registered [ 68.987782] Key type .llcrypt registered [ 69.706935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 69.716489] alg: No test for adler32 (adler32-zlib) [ 71.206858] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 72.391701] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 74.187432] Key type lgssc registered [ 74.816062] hrtimer: interrupt took 2264626 ns [ 76.514044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 214.953356] Lustre: Mounted lustre-client [ 221.360568] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 239.509337] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing check_logdir /tmp/testlogs/ [ 240.608212] Lustre: lustre-OST0000-osc-ffff9ef1c3e63000: disconnect after 23s idle [ 246.519390] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing yml_node [ 252.319161] Lustre: DEBUG MARKER: Client: 2.17.54.105 [ 255.879396] Lustre: DEBUG MARKER: MDS: 2.17.54.105 [ 258.930356] Lustre: DEBUG MARKER: OSS: 2.17.54.105 [ 261.250426] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Jun 27 20:59:54 EDT 2026 [ 280.393202] Lustre: Unmounted lustre-client [ 319.431505] Key type lgssc unregistered [ 319.698555] LNet: 8408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.707030] LNetError: 8408:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 319.736276] LNet: Removed LNI 192.168.201.43@tcp [ 320.951204] Key type .llcrypt unregistered [ 320.953769] Key type ._llcrypt unregistered [ 349.353422] Key type ._llcrypt registered [ 349.355796] Key type .llcrypt registered [ 349.417545] LNet: 9001:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 349.417545] mask usage: [+|-] ... [ 349.627722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 349.664222] alg: No test for adler32 (adler32-zlib) [ 350.670229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 350.752702] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 350.774439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 355.026795] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 355.270851] LNet: 9307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.276462] LNetError: 9307:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 355.294813] LNet: Removed LNI 192.168.201.43@tcp [ 355.854448] Key type .llcrypt unregistered [ 355.857173] Key type ._llcrypt unregistered [ 363.036705] Key type ._llcrypt registered [ 363.039600] Key type .llcrypt registered [ 364.336992] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:01:38 (1782608498) [ 364.799414] LNet: 9680:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 364.799414] mask usage: [+|-] ... [ 365.002354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 365.022168] alg: No test for adler32 (adler32-zlib) [ 365.985676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.089267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.110891] LNet: 9718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.137410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.239394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 366.271704] LNet: 9725:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 373.643927] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 21:01:47 (1782608507) [ 373.933029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.955126] LNet: 10303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.975676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.089254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 380.885382] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 21:01:55 (1782608515) [ 381.106964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.126549] LNet: 10889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.148954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.253244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 388.163651] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 21:02:02 (1782608522) [ 388.468716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.498054] LNet: 11474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.541329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.601153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 395.572163] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 21:02:09 (1782608529) [ 395.733805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.808072] LNet: 12055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.862403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.954814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 396.565019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 403.116503] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 21:02:17 (1782608537) [ 403.355470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.380801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.435531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 403.722684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 403.981549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 411.885865] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 21:02:25 (1782608545) [ 412.225259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.269488] LNet: 13247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.272768] LNet: 13247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 412.337784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.511817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 422.230109] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 21:02:35 (1782608555) [ 422.393038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.463759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.592503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 422.678665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 423.400734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 423.428615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 423.869229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 423.940905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 424.929183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 424.980797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 425.435860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 425.489329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 426.167789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 426.230373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 427.105446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 427.157258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 427.602491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 427.646512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 428.461538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 428.548774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 429.351304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 429.423894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 430.211496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 430.269794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 431.134537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 431.195996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 431.942392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 432.026332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 440.967460] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 21:02:54 (1782608574) [ 441.077982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.160162] LNet: 14508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 441.167528] LNet: 14508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 441.198084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.292131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 441.339363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 448.020418] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 21:03:01 (1782608581) [ 448.122991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.166519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.231975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 448.297163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 455.647160] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 21:03:09 (1782608589) [ 456.015917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.072224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.202472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 456.259933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 464.136481] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 21:03:17 (1782608597) [ 464.255430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.316446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.452124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 464.496757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 472.485701] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 21:03:26 (1782608606) [ 472.599457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.658898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.811990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 472.892327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 480.146717] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 21:03:34 (1782608614) [ 480.336671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.368870] LNet: 17448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 480.379634] LNet: 17448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 480.425650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.562775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 480.636644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 488.524202] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 21:03:42 (1782608622) [ 488.802601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.854915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.950424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 489.354628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 496.023171] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 21:03:49 (1782608629) [ 496.119348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.161056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.363455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 496.561986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 503.421230] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 21:03:57 (1782608637) [ 503.664955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.751160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.926588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 504.180178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 510.732522] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 21:04:05 (1782608645) [ 511.074539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.110529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.770489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 512.078433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 518.530880] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:04:12 (1782608652) [ 518.636275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.681926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.764544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 518.841436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 525.636521] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:04:19 (1782608659) [ 525.990937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.036818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.100125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 526.195796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 534.414312] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 21:04:28 (1782608668) [ 534.726412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.766295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.862543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 540.982674] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:04:35 (1782608675) [ 541.267356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.305570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.386367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 547.923397] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 21:04:41 (1782608681) [ 548.145727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.162879] LNet: 22886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 548.169129] LNet: 22886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 548.185831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.283588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 548.904479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 556.433970] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 21:04:50 (1782608690) [ 556.633959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.695162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.760207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 556.776270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 563.094468] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 21:04:57 (1782608697) [ 563.349796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.387846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.462684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 563.510878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 570.729731] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 21:05:04 (1782608704) [ 570.927062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.016877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.145159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 571.175755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 578.224123] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 21:05:11 (1782608711) [ 578.361941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.415470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.488216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 578.526217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 587.311428] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:05:20 (1782608720) [ 587.407223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.467707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.549083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 587.622949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 587.651096] LNetError: 25836:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 587.709542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 587.777331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 595.736415] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:05:29 (1782608729) [ 596.029346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.089623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.185933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.JPoAxg0Oxp [ 596.272815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.JPoAxg0Oxp [ 596.348255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.JPoAxg0Oxp [ 604.170692] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 21:05:38 (1782608738) [ 604.547303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.606128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.683327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 604.737407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 604.785633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 612.728796] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 21:05:46 (1782608746) [ 612.854059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.891397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.973202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 616.098894] LNet: Message forwarding will be enabled in 2s [ 619.168140] LNet: Message forwarding enabled [ 619.874429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 619.898172] LNet: Message forwarding will stop in 2s to 6s [ 623.449699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 624.398492] LNet: Message forwarding disabled [ 625.915273] LNet: Message forwarding will be enabled in 2s [ 629.536150] LNet: Message forwarding enabled [ 633.990425] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 21:06:07 (1782608767) [ 634.189534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.689952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.821498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 642.556501] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:06:16 (1782608776) [ 642.722608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.803565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.846547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 642.876567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 642.914540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 642.964261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 643.008513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 643.048675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 643.103678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 643.150279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 643.191805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 643.223824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 643.262418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 643.293672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 643.335894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 643.379531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 643.424799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 643.494394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 643.539384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 643.571812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 643.599804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 643.614114] LNetError: 28899:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 650.374907] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:06:23 (1782608783) [ 650.674556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.720987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.802479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 658.583598] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 21:06:32 (1782608792) [ 660.439426] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg143-server [ 662.358131] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 21:06:36 (1782608796) [ 662.472574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.543549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.640949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 662.667876] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 665.101354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[44-48]@tcp [ 665.133858] LNet: 30616:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 665.171751] LNetError: 30406:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.46@tcp is being used as a gateway but routing feature is not turned on [ 665.187130] LNetError: 30406:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.46@tcp has gone from up to down [ 666.222350] LNetError: 30406:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.45@tcp is being used as a gateway but routing feature is not turned on [ 666.232940] LNetError: 30406:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 667.255164] LNetError: 30406:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.45@tcp is being used as a gateway but routing feature is not turned on [ 667.273310] LNetError: 30406:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 669.294449] LNetError: 30406:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.44@tcp is being used as a gateway but routing feature is not turned on [ 669.317992] LNetError: 30406:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 672.664465] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 21:06:46 (1782608806) [ 673.459706] LNetError: 30406:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.45@tcp is being used as a gateway but routing feature is not turned on [ 673.472745] LNetError: 30406:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 674.684203] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg143-server [ 676.240386] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 21:06:50 (1782608810) [ 676.375440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.415343] LNet: 31545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.424651] LNet: 31545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 676.431624] LNetError: 31545:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.452283] LNet: Removed LNI 192.168.201.43@tcp [ 676.522944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.586969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 676.605976] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 677.205051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[44-49/2]@tcp [ 677.240271] LNet: 31668:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 677.253547] LNet: 31668:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 677.280757] LNetError: 31555:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.44@tcp has gone from up to down [ 677.293039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[44-49/2]@tcp [ 677.312359] LNetError: 31555:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 683.804178] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 21:06:57 (1782608817) [ 683.892826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.913541] LNetError: 32255:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 683.936822] LNet: Removed LNI 192.168.201.43@tcp [ 683.983941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.084922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 684.172781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.240174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.298432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 684.359969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.416392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.473611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 684.557054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.596184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.666649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 684.750286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.784893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.823993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 684.913773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.950277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.018068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 685.083337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.117675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.161832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 685.238524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.278844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.388267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 685.542385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.620447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.701348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 685.819128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.916883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.008963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 686.062760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.097707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.148411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 686.202201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.230710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.318801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 692.014689] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:07:06 (1782608826) [ 694.408919] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg143-server [ 696.377978] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 21:07:10 (1782608830) [ 698.212689] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg143-server [ 699.865747] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 21:07:14 (1782608834) [ 700.494617] Key type .llcrypt unregistered [ 700.496288] Key type ._llcrypt unregistered [ 706.603684] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 706.755742] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 706.768417] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 706.781994] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 709.465931] Key type ._llcrypt registered [ 709.468420] Key type .llcrypt registered [ 709.507371] LNet: 33941:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 709.507371] mask usage: [+|-] ... [ 709.666762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 709.683543] alg: No test for adler32 (adler32-zlib) [ 710.635936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.692042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 710.711701] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 711.347827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 711.364217] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 713.075908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 713.090561] LNet: Removed LNI 10.1.2.3@tcp [ 713.178873] LNet: 34382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.183641] LNetError: 34382:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 713.198209] LNet: Removed LNI 192.168.201.43@tcp [ 713.618719] Key type .llcrypt unregistered [ 713.623676] Key type ._llcrypt unregistered [ 724.145367] Key type ._llcrypt registered [ 724.153752] Key type .llcrypt registered [ 725.710241] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 21:07:39 (1782608859) [ 726.179794] Key type .llcrypt unregistered [ 726.182128] Key type ._llcrypt unregistered [ 732.094352] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 732.268258] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 732.276340] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 732.285089] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 734.934990] Key type ._llcrypt registered [ 734.936750] Key type .llcrypt registered [ 734.969995] LNet: 35420:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 734.969995] mask usage: [+|-] ... [ 735.118149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 735.133664] alg: No test for adler32 (adler32-zlib) [ 736.051607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.115228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 736.143567] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 736.688453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 736.705995] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 738.921478] LNet: 35878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.925833] LNetError: 35878:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 738.930108] LNet: Removed LNI 10.1.2.3@tcp [ 738.940745] LNet: Removed LNI 192.168.201.43@tcp [ 739.471548] Key type .llcrypt unregistered [ 739.474719] Key type ._llcrypt unregistered [ 750.602429] Key type ._llcrypt registered [ 750.604691] Key type .llcrypt registered [ 751.916548] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 21:08:05 (1782608885) [ 752.417646] Key type .llcrypt unregistered [ 752.419985] Key type ._llcrypt unregistered [ 758.679918] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 758.779096] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 758.783674] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 758.797730] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 761.497923] Key type ._llcrypt registered [ 761.500397] Key type .llcrypt registered [ 761.544583] LNet: 36913:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 761.544583] mask usage: [+|-] ... [ 761.744674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 761.770568] alg: No test for adler32 (adler32-zlib) [ 762.719336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.788481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 762.814512] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 764.673615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 764.703874] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 767.141164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 767.180246] LNet: Removed LNI 10.1.2.3@tcp [ 767.195871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 767.229447] LNet: Removed LNI 10.1.2.31@tcp [ 767.268756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 767.331187] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 768.165354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 768.201325] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 769.174079] LNet: 37797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.185989] LNetError: 37797:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 769.196057] LNet: Removed LNI 10.1.2.31@tcp [ 769.212424] LNet: Removed LNI 10.1.2.3@tcp [ 769.855437] Key type .llcrypt unregistered [ 769.857420] Key type ._llcrypt unregistered [ 783.539815] Key type ._llcrypt registered [ 783.542566] Key type .llcrypt registered [ 785.056802] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:08:38 (1782608918) [ 785.644848] LNet: 38591:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 785.644848] mask usage: [+|-] ... [ 785.817722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 785.836141] alg: No test for adler32 (adler32-zlib) [ 786.795864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.876202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 786.907420] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 787.857927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.879587] LNet: 38751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.888271] LNetError: 38751:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.907271] LNet: Removed LNI 192.168.201.43@tcp [ 787.936476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.020507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.43@tcp [ 788.042363] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 789.107172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.43@tcp [ 789.171261] LNet: Removed LNI 192.168.201.43@tcp [ 789.247748] LNet: 38886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.046431] Key type .llcrypt unregistered [ 790.052548] Key type ._llcrypt unregistered [ 800.892511] Key type ._llcrypt registered [ 800.894543] Key type .llcrypt registered [ 802.013284] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:08:56 (1782608936) [ 804.354633] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg143-server [ 806.525533] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:09:00 (1782608940) [ 807.146364] Key type .llcrypt unregistered [ 807.148773] Key type ._llcrypt unregistered [ 814.188700] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 814.510616] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 814.557542] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 814.575922] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 817.222941] Key type ._llcrypt registered [ 817.225965] Key type .llcrypt registered [ 817.305212] LNet: 40264:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 817.305212] mask usage: [+|-] ... [ 817.557223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 817.578631] alg: No test for adler32 (adler32-zlib) [ 818.581747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.643825] LNet: Added LNI 192.168.201.43@tcp1 [8/256/0/180] [ 819.404969] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 821.541140] LNet: 40724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.545823] LNetError: 40724:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.551473] LNet: Removed LNI 192.168.201.43@tcp1 [ 821.564247] LNet: Removed LNI 10.1.2.3@tcp2 [ 822.408292] Key type .llcrypt unregistered [ 822.411293] Key type ._llcrypt unregistered [ 836.784343] Key type ._llcrypt registered [ 836.786400] Key type .llcrypt registered [ 838.489426] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:09:32 (1782608972) [ 839.085591] LNet: 41530:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 839.085591] mask usage: [+|-] ... [ 839.356243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 839.374233] alg: No test for adler32 (adler32-zlib) [ 840.384145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.456806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 840.492870] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 841.903624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.951104] LNet: 41696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.958505] LNetError: 41696:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.986139] LNet: Removed LNI 192.168.201.43@tcp [ 842.053864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.178717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 842.215247] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 851.803347] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:09:45 (1782608985) [ 852.220498] LNet: 42289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.257701] LNetError: 42289:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.288409] LNet: Removed LNI 192.168.201.43@tcp [ 853.377349] Key type .llcrypt unregistered [ 853.379621] Key type ._llcrypt unregistered [ 860.908915] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 861.039155] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 861.054055] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 861.068915] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 863.737949] Key type ._llcrypt registered [ 863.740869] Key type .llcrypt registered [ 863.797949] LNet: 42568:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 863.797949] mask usage: [+|-] ... [ 864.013111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 864.027824] alg: No test for adler32 (adler32-zlib) [ 864.983774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.051310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 865.080339] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 866.044928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 866.077186] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 868.300894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.309573] LNet: 43034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.312844] LNetError: 43034:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 868.315931] LNet: Removed LNI 10.1.2.3@tcp [ 868.323575] LNet: Removed LNI 192.168.201.43@tcp [ 868.334798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.382320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 868.402336] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 868.405639] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 868.746299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.759501] LNet: Removed LNI 10.1.2.3@tcp [ 868.768619] LNet: Removed LNI 192.168.201.43@tcp [ 868.778517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.807731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 868.826349] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 868.830324] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 869.114543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 869.129840] LNet: Added LNI 192.168.201.43@tcp2 [8/256/0/180] [ 870.300415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.339946] LNet: 43224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.353302] LNet: 43224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 870.360172] LNetError: 43224:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 870.366245] LNetError: 43224:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 870.379347] LNet: Removed LNI 10.1.2.3@tcp [ 870.388782] LNet: Removed LNI 192.168.201.43@tcp [ 870.411435] LNet: Removed LNI 192.168.201.43@tcp2 [ 870.467397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.562766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 870.594309] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 870.599997] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 870.605399] LNet: Added LNI 192.168.201.43@tcp2 [8/256/0/180] [ 878.158581] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:10:12 (1782609012) [ 878.290156] LNet: 43815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.298111] LNetError: 43815:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 878.303381] LNetError: 43815:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 878.310590] LNet: Removed LNI 10.1.2.3@tcp [ 878.313996] LNet: Removed LNI 192.168.201.43@tcp [ 878.347480] LNet: Removed LNI 192.168.201.43@tcp2 [ 879.138576] Key type .llcrypt unregistered [ 879.141156] Key type ._llcrypt unregistered [ 884.806565] Key type ._llcrypt registered [ 884.810338] Key type .llcrypt registered [ 884.862307] LNet: 44048:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 884.862307] mask usage: [+|-] ... [ 884.966984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 884.981220] alg: No test for adler32 (adler32-zlib) [ 885.961088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.008175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 886.032547] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 886.901634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 886.926409] LNet: 44218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.933709] LNetError: 44218:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.966688] LNet: Removed LNI 192.168.201.43@tcp [ 887.014813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.123613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 887.157352] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 887.730629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 887.746927] LNet: 44248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.751793] LNetError: 44248:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 887.774290] LNet: Removed LNI 192.168.201.43@tcp [ 887.806675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.895678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 887.925616] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 888.427852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 888.458837] LNet: Removed LNI 192.168.201.43@tcp [ 888.476347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.536900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 888.587845] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 889.057531] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 889.222620] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 889.226771] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 889.236022] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 891.736681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.764583] LNet: 44357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.770372] LNet: 44357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 891.776057] LNetError: 44357:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.779348] LNetError: 44357:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 891.812648] LNet: Removed LNI 192.168.201.43@tcp [ 891.840205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.908544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 891.946163] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 898.096332] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:10:32 (1782609032) [ 898.248570] LNet: 44954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.261475] LNetError: 44954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.283214] LNet: Removed LNI 192.168.201.43@tcp [ 898.857310] Key type .llcrypt unregistered [ 898.858828] Key type ._llcrypt unregistered [ 905.375317] Key type ._llcrypt registered [ 905.382117] Key type .llcrypt registered [ 905.428802] LNet: 45181:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 905.428802] mask usage: [+|-] ... [ 905.596269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 905.609753] alg: No test for adler32 (adler32-zlib) [ 906.633945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.733466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 906.766878] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 907.738514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.759751] LNet: 45355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.767585] LNetError: 45355:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.783512] LNet: Removed LNI 192.168.201.43@tcp [ 907.801820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.877610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 907.909910] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 915.472200] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:10:49 (1782609049) [ 915.538612] LNet: 45940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.544668] LNetError: 45940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.572658] LNet: Removed LNI 192.168.201.43@tcp [ 916.070456] Key type .llcrypt unregistered [ 916.072234] Key type ._llcrypt unregistered [ 921.604455] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 921.688622] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 921.690906] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 921.703802] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 924.403359] Key type ._llcrypt registered [ 924.405607] Key type .llcrypt registered [ 924.455052] LNet: 46229:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 924.455052] mask usage: [+|-] ... [ 924.642933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 924.655531] alg: No test for adler32 (adler32-zlib) [ 925.580370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.637739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 925.652443] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 926.339713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.359584] LNet: 46402:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.367683] LNetError: 46402:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.384863] LNet: Removed LNI 192.168.201.43@tcp [ 926.417864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.462764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 926.487928] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 932.631516] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:11:06 (1782609066) [ 932.927350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.968376] LNet: 47001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.987690] LNetError: 47001:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.006535] LNet: Removed LNI 192.168.201.43@tcp [ 933.030907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.159771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 940.139694] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:11:14 (1782609074) [ 940.753226] LNet: 47583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.533806] Key type .llcrypt unregistered [ 941.537601] Key type ._llcrypt unregistered [ 948.873402] Key type ._llcrypt registered [ 948.875245] Key type .llcrypt registered [ 948.907734] LNet: 47809:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 948.907734] mask usage: [+|-] ... [ 949.071553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 949.100627] alg: No test for adler32 (adler32-zlib) [ 950.117205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.237442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 950.311706] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 951.229581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.282527] LNet: 47981:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.288878] LNetError: 47981:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.310308] LNet: Removed LNI 192.168.201.43@tcp [ 951.334772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.406907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 951.440392] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 958.241173] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:11:32 (1782609092) [ 958.552331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.570476] LNet: 48574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.573827] LNetError: 48574:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.604703] LNet: Removed LNI 192.168.201.43@tcp [ 958.618369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.659299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 963.610330] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:11:38 (1782609098) [ 965.932582] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 968.362891] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:11:41 (1782609101) [ 970.718816] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 972.652500] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:11:46 (1782609106) [ 975.149814] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 977.378264] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:11:51 (1782609111) [ 979.181525] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 981.448462] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:11:55 (1782609115) [ 983.019425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 984.900900] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 21:11:58 (1782609118) [ 985.210589] LNet: 50626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.810389] Key type .llcrypt unregistered [ 985.813513] Key type ._llcrypt unregistered [ 993.154206] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 993.301287] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 993.303771] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 993.306824] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 996.035555] Key type ._llcrypt registered [ 996.037344] Key type .llcrypt registered [ 996.068559] LNet: 50906:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 996.068559] mask usage: [+|-] ... [ 996.290891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 996.307895] alg: No test for adler32 (adler32-zlib) [ 997.246088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.317256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 997.338284] LNet: Added LNI 192.168.201.43@tcp [8/512/0/180] [ 997.350858] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 997.353659] LNet: 50944:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 997.449606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.466696] LNet: 50964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.476187] LNetError: 50964:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 997.484382] LNet: Removed LNI 10.1.2.3@tcp [ 997.486051] LNetError: 50964:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.502313] LNet: Removed LNI 192.168.201.43@tcp [ 997.537423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.611266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 997.635724] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 997.729174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.775603] LNet: 50988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.782874] LNetError: 50988:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.797148] LNet: Removed LNI 192.168.201.43@tcp [ 997.819522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.900613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 997.926431] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 997.933669] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 997.941095] LNet: 50996:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 998.027410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.058297] LNet: 51014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.065228] LNetError: 51014:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 998.070605] LNet: Removed LNI 10.1.2.3@tcp [ 998.076534] LNetError: 51014:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.089907] LNet: Removed LNI 192.168.201.43@tcp [ 998.127446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.221589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 998.246095] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 998.250444] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 998.253285] LNet: 51022:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 998.451644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.474479] LNet: 51048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.481767] LNetError: 51048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 998.486497] LNet: Removed LNI 10.1.2.3@tcp [ 998.489595] LNetError: 51048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.524113] LNet: Removed LNI 192.168.201.43@tcp [ 998.539898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.615757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 998.638271] LNet: Added LNI 192.168.201.43@tcp [8/256/0/182] [ 998.641396] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 998.719644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.751896] LNet: 51076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.761749] LNetError: 51076:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 998.767936] LNet: Removed LNI 10.1.2.3@tcp [ 998.770868] LNetError: 51076:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.785504] LNet: Removed LNI 192.168.201.43@tcp [ 998.814104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.909037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 998.943427] LNet: Added LNI 192.168.201.43@tcp [8/256/0/182] [ 998.947853] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 998.949343] LNet: 51084:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 999.062941] LNet: Removed LNI 10.1.2.3@tcp [ 999.099696] LNet: Removed LNI 192.168.201.43@tcp [ 999.989544] Key type .llcrypt unregistered [ 999.992574] Key type ._llcrypt unregistered [ 1012.555989] Key type ._llcrypt registered [ 1012.558111] Key type .llcrypt registered [ 1014.088467] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:12:27 (1782609147) [ 1014.522985] LNet: 51898:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1014.522985] mask usage: [+|-] ... [ 1014.750840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1014.779503] alg: No test for adler32 (adler32-zlib) [ 1015.739067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.856798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1015.899544] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1016.526786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.43 [ 1016.543688] LNet: 52059:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1016.548526] LNet: Removed LNI 192.168.201.43@tcp [ 1022.734361] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:12:36 (1782609156) [ 1022.874093] LNet: 52623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.885718] LNetError: 52623:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.906073] LNet: Removed LNI 192.168.201.43@tcp [ 1023.774567] Key type .llcrypt unregistered [ 1023.776540] Key type ._llcrypt unregistered [ 1028.905634] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1029.202867] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1031.927343] Key type ._llcrypt registered [ 1031.934659] Key type .llcrypt registered [ 1031.974932] LNet: 52911:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1031.974932] mask usage: [+|-] ... [ 1032.107921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1032.118994] alg: No test for adler32 (adler32-zlib) [ 1033.559588] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1040.284404] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:12:53 (1782609173) [ 1040.405053] LNet: 53534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.419312] LNetError: 53534:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1040.456439] LNet: Removed LNI 10.1.2.3@tcp [ 1041.028741] Key type .llcrypt unregistered [ 1041.031460] Key type ._llcrypt unregistered [ 1046.922622] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1047.148510] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1047.153706] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1047.162718] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1050.029159] Key type ._llcrypt registered [ 1050.044496] Key type .llcrypt registered [ 1050.100141] LNet: 53816:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1050.100141] mask usage: [+|-] ... [ 1050.259610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1050.276978] alg: No test for adler32 (adler32-zlib) [ 1051.315062] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1058.226747] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:13:12 (1782609192) [ 1060.185490] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1061.819600] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:13:16 (1782609196) [ 1062.257118] LNet: 54727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.266374] LNetError: 54727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1062.279242] LNet: Removed LNI 10.1.2.3@tcp [ 1063.171572] Key type .llcrypt unregistered [ 1063.173824] Key type ._llcrypt unregistered [ 1069.581130] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1069.822474] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1069.832040] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1069.840600] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1072.761617] Key type ._llcrypt registered [ 1072.762929] Key type .llcrypt registered [ 1072.791853] LNet: 55001:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1072.791853] mask usage: [+|-] ... [ 1072.923730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1072.937090] alg: No test for adler32 (adler32-zlib) [ 1073.890077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1073.975746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1082.193045] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:13:36 (1782609216) [ 1082.830914] LNet: 55958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.837378] LNetError: 55958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1082.864984] LNet: Removed LNI 10.1.2.3@tcp [ 1083.495746] Key type .llcrypt unregistered [ 1083.499524] Key type ._llcrypt unregistered [ 1089.210225] Key type ._llcrypt registered [ 1089.217216] Key type .llcrypt registered [ 1089.278267] LNet: 56181:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1089.278267] mask usage: [+|-] ... [ 1089.467599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1089.494110] alg: No test for adler32 (adler32-zlib) [ 1090.483978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1090.519833] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1099.314207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.143@tcp [ 1100.453960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_interrupt -r 1 [ 1100.489386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_interrupt -r 1 [ 1100.527735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1100.539354] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1100.769169] LNet: 1 local NIs in recovery (showing 1): 192.168.201.43@tcp [ 1100.869416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1100.914622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1100.969417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1101.020053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1101.088677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1101.153698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1101.215456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1101.290641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1101.315456] LNet: 56395:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1101.334758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1101.384578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1101.427816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1101.482707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1101.524233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1101.574903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1101.630673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1102.406311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_dropped -r 1 [ 1102.489981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_dropped -r 1 [ 1102.542713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1102.580127] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1102.865778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1102.888387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1102.904967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1102.922748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.951814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1102.983181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1103.015786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1103.056608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1103.078258] LNet: 56505:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1103.099390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1103.148956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1103.233723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1103.302688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1103.360790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1103.395434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1103.429806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1104.072284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_aborted -r 1 [ 1104.158743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_aborted -r 1 [ 1104.240646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1104.283530] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1104.636482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.667379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1104.694414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.723214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.748484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.771293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.810888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.918526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.959523] LNet: 56614:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1105.018573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.087752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.157666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.221104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.312387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.364714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.410670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1105.948088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_no_route -r 1 [ 1105.987568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_no_route -r 1 [ 1106.015728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1106.355892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.418878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1106.488770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.540722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.618656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.698093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1106.760993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1106.782916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.805766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.826202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.842766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.862239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.893742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.944700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.971772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1107.527873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_timeout -r 1 [ 1107.581481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_timeout -r 1 [ 1107.620049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1107.636730] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1107.646418] LNet: Skipped 1 previous similar message [ 1107.928053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.956928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1107.980857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.998518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.023350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.060649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.100837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.136952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.153629] LNet: 56832:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.158461] LNet: 56832:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1108.173054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.209824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.266595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.302224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.335701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.399528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.486646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1109.038922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_error -r 1 [ 1109.072707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_error -r 1 [ 1109.117914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1109.464465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.493475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1109.517584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.536802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.557153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.588306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.623899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.671369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.700812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.742410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.784306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.807779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.837792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.879189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.944873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1110.349902] LNet: 56983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.360144] LNetError: 56983:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1110.388511] LNet: Removed LNI 192.168.201.43@tcp [ 1110.804576] Key type .llcrypt unregistered [ 1110.808208] Key type ._llcrypt unregistered [ 1130.707209] Key type ._llcrypt registered [ 1130.710469] Key type .llcrypt registered [ 1132.414227] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:14:25 (1782609265) [ 1133.657701] Key type .llcrypt unregistered [ 1133.662605] Key type ._llcrypt unregistered [ 1141.399802] Key type ._llcrypt registered [ 1141.403153] Key type .llcrypt registered [ 1141.489601] LNet: 58021:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1141.489601] mask usage: [+|-] ... [ 1141.740684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1141.758721] alg: No test for adler32 (adler32-zlib) [ 1142.748866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1142.798345] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1150.760555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.143@tcp [ 1162.145543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1162.193678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1162.209281] LNet: Added LNI 192.168.201.43@tcp1 [8/256/0/180] [ 1164.247271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_interrupt -r 1 [ 1164.288900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp1 -m GET -e local_interrupt -r 1 [ 1164.327724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_interrupt -r 1 [ 1164.368163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -m GET -e local_interrupt -r 1 [ 1164.418741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp -m GET -e local_interrupt -r 1 [ 1164.448587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp1 -m GET -e local_interrupt -r 1 [ 1164.486496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -m GET -e local_interrupt -r 1 [ 1164.527412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -m GET -e local_interrupt -r 1 [ 1164.578511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1164.600279] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1164.603156] LNet: 1 local NIs in recovery (showing 1): 192.168.201.43@tcp [ 1164.881620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.938455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1164.999982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.060733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.097310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.155411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.220106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.289409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.329747] LNet: 58451:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1165.354289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.388949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.422554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.484558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.527950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.590383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.635659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1166.481721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_dropped -r 1 [ 1166.526630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp1 -m GET -e local_dropped -r 1 [ 1166.559911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_dropped -r 1 [ 1166.607195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -m GET -e local_dropped -r 1 [ 1166.681746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp -m GET -e local_dropped -r 1 [ 1166.786546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp1 -m GET -e local_dropped -r 1 [ 1166.835070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -m GET -e local_dropped -r 1 [ 1166.875531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -m GET -e local_dropped -r 1 [ 1166.918591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1166.944752] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1166.947720] LNet: Skipped 2 previous similar messages [ 1167.235561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.280996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1167.323819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.381971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.430401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.493974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.546306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.583779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.611664] LNet: 58572:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1167.646890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.692255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.731779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.794414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.850225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.927497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.986266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1168.532725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_aborted -r 1 [ 1168.554388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp1 -m GET -e local_aborted -r 1 [ 1168.577118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_aborted -r 1 [ 1168.621640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -m GET -e local_aborted -r 1 [ 1168.701669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp -m GET -e local_aborted -r 1 [ 1168.756799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp1 -m GET -e local_aborted -r 1 [ 1168.802648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -m GET -e local_aborted -r 1 [ 1168.841560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -m GET -e local_aborted -r 1 [ 1168.873983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1168.893580] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1168.899683] LNet: Skipped 2 previous similar messages [ 1169.194914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.238172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1169.305870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.359734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1169.386230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1169.419604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1169.448082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1169.478569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1169.501375] LNet: 58694:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1169.521493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1169.575616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1169.616962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1169.655755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1169.731807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.770752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.841499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1170.447638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_no_route -r 1 [ 1170.485935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp1 -m GET -e local_no_route -r 1 [ 1170.514990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_no_route -r 1 [ 1170.547888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -m GET -e local_no_route -r 1 [ 1170.574305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp -m GET -e local_no_route -r 1 [ 1170.616525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp1 -m GET -e local_no_route -r 1 [ 1170.659342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -m GET -e local_no_route -r 1 [ 1170.687753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -m GET -e local_no_route -r 1 [ 1170.712853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1170.914884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.943186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1170.967452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.002545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.042910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.083933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.115754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.154755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.190342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.222251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.255944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.283158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.359890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.437188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.480888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.046770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_timeout -r 1 [ 1172.083830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp1 -m GET -e local_timeout -r 1 [ 1172.132841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_timeout -r 1 [ 1172.173180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -m GET -e local_timeout -r 1 [ 1172.228459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp -m GET -e local_timeout -r 1 [ 1172.299315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp1 -m GET -e local_timeout -r 1 [ 1172.359732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -m GET -e local_timeout -r 1 [ 1172.422933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -m GET -e local_timeout -r 1 [ 1172.473359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1172.508418] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1172.516465] LNet: Skipped 5 previous similar messages [ 1172.801655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.837521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1172.866487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1172.915983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1172.957626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.007909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.044442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.086447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.106297] LNet: 58936:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.112136] LNet: 58936:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1173.131568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.173824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.204970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.223950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.253174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.285595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.319651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1173.777849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_error -r 1 [ 1173.814994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp1 -m GET -e local_error -r 1 [ 1173.851756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_error -r 1 [ 1173.882776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -m GET -e local_error -r 1 [ 1173.938356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp -m GET -e local_error -r 1 [ 1174.034989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp1 -m GET -e local_error -r 1 [ 1174.087730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -m GET -e local_error -r 1 [ 1174.148936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -m GET -e local_error -r 1 [ 1174.201473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1174.451225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.488304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1174.517972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.549802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.599448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.632782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.668842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.702914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.737097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.781469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.811914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.847337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.884471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.914206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.942529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1180.502985] LNet: 59133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1180.513542] LNetError: 59133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1180.519754] LNet: Removed LNI 192.168.201.43@tcp [ 1180.529577] LNet: Removed LNI 192.168.201.43@tcp1 [ 1180.929739] Key type .llcrypt unregistered [ 1180.932256] Key type ._llcrypt unregistered [ 1200.878852] Key type ._llcrypt registered [ 1200.881403] Key type .llcrypt registered [ 1203.199049] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:15:36 (1782609336) [ 1204.573675] Key type .llcrypt unregistered [ 1204.576666] Key type ._llcrypt unregistered [ 1213.203582] Key type ._llcrypt registered [ 1213.207991] Key type .llcrypt registered [ 1213.261204] LNet: 60170:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1213.261204] mask usage: [+|-] ... [ 1213.465716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1213.498185] alg: No test for adler32 (adler32-zlib) [ 1214.547642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1214.597445] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1223.525868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.143@tcp [ 1224.604262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e remote_dropped -r 1 [ 1224.650888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e remote_dropped -r 1 [ 1224.684203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1224.705830] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1224.962441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1224.996257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1225.030946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1225.066755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1225.102653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1225.160119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1225.208394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1225.266470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1225.301737] LNet: 60385:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1225.342733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1225.418897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1225.522636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1225.595663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1225.653459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1225.726889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1225.780824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1226.632412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e remote_error -r 1 [ 1226.705806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e remote_error -r 1 [ 1226.774735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1226.800281] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1227.025777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1227.052846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1227.086639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1227.116200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1227.146192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1227.187859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1227.228246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1227.269710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1227.293080] LNet: 60503:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1227.320704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1227.405034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1227.441796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1227.457874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1227.492108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.537631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1227.576630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1228.254650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e remote_timeout -r 1 [ 1228.295901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e remote_timeout -r 1 [ 1228.342435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1228.370703] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1228.637175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1228.686436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1228.731193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.775233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.813430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.851796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.893720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1228.978312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1229.010158] LNet: 60612:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1229.051576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1229.095649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1229.172843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1229.255856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1229.320140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1229.349287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1229.386890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1229.645254] LNet: 60654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1229.654116] LNetError: 60654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1230.710167] LNet: Removed LNI 192.168.201.43@tcp [ 1231.251753] Key type .llcrypt unregistered [ 1231.258130] Key type ._llcrypt unregistered [ 1248.767597] Key type ._llcrypt registered [ 1248.771670] Key type .llcrypt registered [ 1250.096240] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:16:24 (1782609384) [ 1251.015599] Key type .llcrypt unregistered [ 1251.018425] Key type ._llcrypt unregistered [ 1256.677608] Key type ._llcrypt registered [ 1256.682436] Key type .llcrypt registered [ 1256.747836] LNet: 61691:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1256.747836] mask usage: [+|-] ... [ 1256.901479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1256.910911] alg: No test for adler32 (adler32-zlib) [ 1257.795186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1257.817961] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1264.757789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.143@tcp [ 1275.345446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1275.412485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1275.450453] LNet: Added LNI 192.168.201.43@tcp1 [8/256/0/180] [ 1276.736503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e remote_dropped -r 1 [ 1276.786523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp1 -m GET -e remote_dropped -r 1 [ 1276.839557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e remote_dropped -r 1 [ 1276.901322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -m GET -e remote_dropped -r 1 [ 1276.969548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp -m GET -e remote_dropped -r 1 [ 1277.004996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp1 -m GET -e remote_dropped -r 1 [ 1277.038340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -m GET -e remote_dropped -r 1 [ 1277.070477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -m GET -e remote_dropped -r 1 [ 1277.103157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1277.120518] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1277.125508] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.143@tcp [ 1277.317456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1277.353380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1277.379660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1277.400401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1277.430230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1277.458900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1277.488210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1277.516110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1277.530062] LNet: 62121:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1277.545521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1277.567850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1277.589087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1277.609979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1277.644622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1277.668936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1277.697484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1277.728420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.276334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e remote_error -r 1 [ 1278.306685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp1 -m GET -e remote_error -r 1 [ 1278.337668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e remote_error -r 1 [ 1278.374335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -m GET -e remote_error -r 1 [ 1278.406482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp -m GET -e remote_error -r 1 [ 1278.446771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp1 -m GET -e remote_error -r 1 [ 1278.474890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -m GET -e remote_error -r 1 [ 1278.503609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -m GET -e remote_error -r 1 [ 1278.539426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1278.569871] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1278.573995] LNet: Skipped 2 previous similar messages [ 1278.826618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1278.848868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1278.881112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1278.907764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1278.936439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1278.962219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.984837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1279.005679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1279.018454] LNet: 62244:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1279.030591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1279.054307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1279.073388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1279.098889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1279.128636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1279.174791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1279.205297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1279.239693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1279.827655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e remote_timeout -r 1 [ 1279.883341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp1 -m GET -e remote_timeout -r 1 [ 1279.926673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e remote_timeout -r 1 [ 1279.959825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -m GET -e remote_timeout -r 1 [ 1280.001366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp -m GET -e remote_timeout -r 1 [ 1280.041726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp1 -m GET -e remote_timeout -r 1 [ 1280.086600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -m GET -e remote_timeout -r 1 [ 1280.155064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -m GET -e remote_timeout -r 1 [ 1280.231849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.143@tcp [ 1280.294751] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1280.539952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1280.563817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1280.588969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1280.613950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1280.642572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.667597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.697444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.726988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.744674] LNet: 62368:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.763628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.794453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.837675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.878312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.926936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.945427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1280.970374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1281.013444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1286.565821] LNet: 62446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1286.577821] LNetError: 62446:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1287.586792] LNet: Removed LNI 192.168.201.43@tcp [ 1287.589757] LNetError: 62446:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1287.609527] LNet: Removed LNI 192.168.201.43@tcp1 [ 1288.065662] Key type .llcrypt unregistered [ 1288.069078] Key type ._llcrypt unregistered [ 1304.117170] Key type ._llcrypt registered [ 1304.120919] Key type .llcrypt registered [ 1305.764746] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:17:19 (1782609439) [ 1307.572459] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1309.682804] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:17:23 (1782609443) [ 1310.592404] Key type .llcrypt unregistered [ 1310.595542] Key type ._llcrypt unregistered [ 1319.012374] Key type ._llcrypt registered [ 1319.016439] Key type .llcrypt registered [ 1319.060990] LNet: 63776:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1319.060990] mask usage: [+|-] ... [ 1319.264402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1319.292260] alg: No test for adler32 (adler32-zlib) [ 1320.368132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1320.414134] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1327.994570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.143@tcp [ 1328.741248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e network_timeout -r 1 [ 1328.782071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e network_timeout -r 1 [ 1328.842703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.143@tcp [ 1328.881635] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1329.111321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1329.159414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1329.192759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1329.236389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1329.273792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1329.309720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1329.345900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1329.360946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1329.369620] LNet: 63991:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1329.378135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1329.392878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1329.407680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1329.421611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1329.437525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1329.452330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1329.567668] LNet: 64030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1329.570615] LNetError: 64030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1329.582474] LNet: Removed LNI 192.168.201.43@tcp [ 1329.792567] Key type .llcrypt unregistered [ 1329.794708] Key type ._llcrypt unregistered [ 1350.580708] Key type ._llcrypt registered [ 1350.584351] Key type .llcrypt registered [ 1350.665273] LNet: 64481:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1350.665273] mask usage: [+|-] ... [ 1350.908113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1350.934944] alg: No test for adler32 (adler32-zlib) [ 1351.920389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1351.970550] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1360.999951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.143@tcp [ 1370.239723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1370.301311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1370.329026] LNet: Added LNI 192.168.201.43@tcp1 [8/256/0/180] [ 1371.921586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e network_timeout -r 1 [ 1371.968670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp1 -m GET -e network_timeout -r 1 [ 1372.016285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e network_timeout -r 1 [ 1372.082201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -m GET -e network_timeout -r 1 [ 1372.195390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp -m GET -e network_timeout -r 1 [ 1372.262951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp1 -m GET -e network_timeout -r 1 [ 1372.304722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -m GET -e network_timeout -r 1 [ 1372.352934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -m GET -e network_timeout -r 1 [ 1372.432537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.143@tcp [ 1372.461409] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1372.807539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1372.860232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1372.899349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1372.950084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1373.000321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1373.034792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1373.066850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1373.110399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1373.129919] LNet: 64912:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1373.152390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1373.189328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1373.251351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1373.313153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1373.366612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1373.418778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1378.279141] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1378.592192] LNet: 1 local NIs in recovery (showing 1): 192.168.201.43@tcp [ 1378.677886] LNet: 64986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.683635] LNetError: 64986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1379.744342] LNet: Removed LNI 192.168.201.43@tcp [ 1379.748671] LNetError: 64986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1379.763212] LNet: Removed LNI 192.168.201.43@tcp1 [ 1380.268650] Key type .llcrypt unregistered [ 1380.270982] Key type ._llcrypt unregistered [ 1396.836531] Key type ._llcrypt registered [ 1396.838581] Key type .llcrypt registered [ 1398.496580] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 21:18:51 (1782609531) [ 1398.801042] LNet: 65786:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1398.801042] mask usage: [+|-] ... [ 1399.104183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1399.152578] alg: No test for adler32 (adler32-zlib) [ 1400.094986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1400.167857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1400.191148] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1401.138551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1401.177304] LNet: Added LNI 192.168.201.43@tcp1 [8/256/0/180] [ 1402.054231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 1402.092991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1402.118766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1402.134521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1402.149202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1402.174325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1402.193321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.43@tcp [ 1402.223164] LNetError: 66087:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.201.43@tcp to recovery [ 1403.232796] LNet: 1 local NIs in recovery (showing 1): 192.168.201.43@tcp [ 1403.233063] LNet: There was an unexpected network error while writing to 192.168.201.43: rc = -22 [ 1405.280465] LNet: There was an unexpected network error while writing to 192.168.201.43: rc = -22 [ 1406.013523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1409.377735] LNet: There was an unexpected network error while writing to 192.168.201.43: rc = -22 [ 1409.827313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1409.873616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1409.953222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1409.973172] LNet: 66195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1409.981666] LNetError: 66195:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1409.991688] LNet: Removed LNI 192.168.201.43@tcp [ 1410.007932] LNet: Removed LNI 192.168.201.43@tcp1 [ 1410.036402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1410.111015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1410.140337] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1411.012974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1411.045858] LNet: Added LNI 192.168.201.43@tcp1 [8/256/0/180] [ 1412.318907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 1412.385267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1412.458954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1412.558695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1412.613507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1412.672470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1412.728240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1412.844702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.43@tcp [ 1412.889626] LNetError: 66467:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.201.43@tcp to recovery [ 1413.152402] LNet: There was an unexpected network error while writing to 192.168.201.43: rc = -22 [ 1415.784730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1419.360525] LNet: There was an unexpected network error while writing to 192.168.201.43: rc = -22 [ 1419.371407] LNet: Skipped 1 previous similar message [ 1424.292414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1424.318984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1424.352963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1430.631360] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 21:19:24 (1782609564) [ 1431.181210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1431.195262] LNet: 67175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.201255] LNetError: 67175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.207143] LNetError: 67175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1431.212050] LNet: Removed LNI 192.168.201.43@tcp [ 1431.221439] LNet: Removed LNI 192.168.201.43@tcp1 [ 1431.240392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1431.304748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1431.337633] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1431.986379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1432.000730] LNet: Added LNI 192.168.201.43@tcp1 [8/256/0/180] [ 1432.907544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 1432.986605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1433.024595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1433.061838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1433.113478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1433.163751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1433.218760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.43@tcp --health 0 [ 1433.316175] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.43@tcp [ 1433.341525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1434.343642] LNet: There was an unexpected network error while writing to 192.168.201.43: rc = -22 [ 1434.354657] LNet: Skipped 1 previous similar message [ 1448.989527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1449.170067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1449.206450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1449.238961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.43@tcp --health 500 [ 1449.322804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1455.890452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1455.978565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1455.989477] LNet: 67666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1455.993695] LNetError: 67666:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1455.996163] LNetError: 67666:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1456.002291] LNet: Removed LNI 192.168.201.43@tcp [ 1456.017439] LNet: Removed LNI 192.168.201.43@tcp1 [ 1456.033365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1456.082315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1456.098919] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1456.722978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1456.737682] LNet: Added LNI 192.168.201.43@tcp1 [8/256/0/180] [ 1457.764995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 1457.826612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1457.903531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1457.933399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1457.977467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1458.029467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1458.093936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1458.134076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.43@tcp --health 0 [ 1459.168457] LNet: There was an unexpected network error while writing to 192.168.201.43: rc = -22 [ 1459.176197] LNet: Skipped 3 previous similar messages [ 1459.530855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1470.235913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1470.283880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1470.312841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1478.848322] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 21:20:12 (1782609612) [ 1479.662457] LNet: 68712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.674854] LNetError: 68712:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1479.682719] LNetError: 68712:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1479.691935] LNet: Removed LNI 192.168.201.43@tcp [ 1479.706638] LNet: Removed LNI 192.168.201.43@tcp1 [ 1480.680291] Key type .llcrypt unregistered [ 1480.692500] Key type ._llcrypt unregistered [ 1487.185383] Key type ._llcrypt registered [ 1487.188754] Key type .llcrypt registered [ 1487.258446] LNet: 68935:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1487.258446] mask usage: [+|-] ... [ 1487.650955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1487.686490] alg: No test for adler32 (adler32-zlib) [ 1488.675984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1488.711408] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1497.936132] Lustre: DEBUG MARKER: Initial discovery [ 1498.011317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.143@tcp [ 1500.243962] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1500.357327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1500.409195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.143@tcp [ 1500.473133] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1500.498659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1500.539871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.143@tcp -r 1 -m GET -l 3 [ 1500.605993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.143@tcp -r 1 -m GET -e local_error [ 1500.652630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.143@tcp -r 1 -m PUT -l 6 [ 1501.027460] LNet: 1 local NIs in recovery (showing 1): 192.168.201.43@tcp [ 1502.066769] Lustre: DEBUG MARKER: Force oleg143-client.virtnet to discover 192.168.201.143@tcp (in background) [ 1502.144026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.143@tcp [ 1504.171688] Lustre: DEBUG MARKER: Force oleg143-server to discover 192.168.201.43@tcp [ 1505.254705] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1506.758975] Lustre: DEBUG MARKER: Wait for 69467 [ 1508.923691] Lustre: DEBUG MARKER: Finished wait on 69467 [ 1508.967388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1509.013030] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1509.020368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1509.604727] LNet: 69929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.623905] LNetError: 69929:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1509.654209] LNet: Removed LNI 192.168.201.43@tcp [ 1510.222381] Key type .llcrypt unregistered [ 1510.225332] Key type ._llcrypt unregistered [ 1531.203513] Key type ._llcrypt registered [ 1531.205030] Key type .llcrypt registered [ 1532.770174] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:21:06 (1782609666) [ 1533.854466] Key type .llcrypt unregistered [ 1533.858529] Key type ._llcrypt unregistered [ 1541.902628] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1542.014388] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1542.016706] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1542.018904] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1544.835604] Key type ._llcrypt registered [ 1544.838249] Key type .llcrypt registered [ 1544.897652] LNet: 70974:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1544.897652] mask usage: [+|-] ... [ 1545.132750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1545.144713] alg: No test for adler32 (adler32-zlib) [ 1546.130496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.188078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1546.246125] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1547.025831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1547.050659] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1556.079666] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:21:29 (1782609689) [ 1556.205168] LNet: 72003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.215482] LNetError: 72003:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1556.221199] LNet: Removed LNI 10.1.2.3@tcp [ 1556.230260] LNet: Removed LNI 192.168.201.43@tcp [ 1557.062432] Key type .llcrypt unregistered [ 1557.064540] Key type ._llcrypt unregistered [ 1564.343347] Key type ._llcrypt registered [ 1564.352628] Key type .llcrypt registered [ 1564.407210] LNet: 72258:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1564.407210] mask usage: [+|-] ... [ 1564.646977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1564.658358] alg: No test for adler32 (adler32-zlib) [ 1565.625578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.686377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1565.720382] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1566.508821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1566.527308] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1567.547244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.43@tcp [ 1568.608406] LNetError: Unexpected error -22 connecting to 192.168.201.43@tcp at host 192.168.201.43:988 [ 1574.339463] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:21:48 (1782609708) [ 1574.443449] LNet: 73133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1574.462281] LNetError: 73133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1575.520161] LNet: Removed LNI 10.1.2.3@tcp [ 1575.521871] LNetError: 73133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.544906] LNet: Removed LNI 192.168.201.43@tcp [ 1576.083262] Key type .llcrypt unregistered [ 1576.085107] Key type ._llcrypt unregistered [ 1582.307633] Key type ._llcrypt registered [ 1582.317279] Key type .llcrypt registered [ 1582.380449] LNet: 73372:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1582.380449] mask usage: [+|-] ... [ 1582.624909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1582.639150] alg: No test for adler32 (adler32-zlib) [ 1583.600822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.663763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1583.703987] LNet: Added LNI 192.168.201.43@tcp1 [8/256/0/180] [ 1584.686839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1584.710648] LNet: Added LNI 192.168.201.43@tcp2 [8/256/0/180] [ 1585.757327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.43@tcp1 --nid 192.168.201.43@tcp2 [ 1592.338461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.43@tcp1 192.168.201.43@tcp2 [ 1592.386333] LNetError: 73976:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.43@tcp1 to 192.168.201.43@tcp2 - Route table may be misconfigured [ 1592.455742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.43@tcp2 192.168.201.43@tcp1 [ 1602.116846] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:22:15 (1782609735) [ 1602.273812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.330908] LNet: 74543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1602.340086] LNetError: 74543:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.360391] LNet: Removed LNI 192.168.201.43@tcp1 [ 1603.365276] LNetError: 74543:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.388205] LNet: Removed LNI 192.168.201.43@tcp2 [ 1603.419385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.482898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1603.516333] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1604.225897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1604.256713] LNet: Added LNI 192.168.201.43@tcp1 [8/256/0/180] [ 1605.037931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 1605.073898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.43@tcp 192.168.201.43@tcp [ 1605.095727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.43@tcp1 192.168.201.43@tcp1 [ 1605.115826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.43@tcp -d 192.168.201.43@tcp -e network_timeout [ 1605.159746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -e network_timeout [ 1605.186777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -e network_timeout [ 1605.228398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -e network_timeout [ 1605.303694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 1605.335729] LNet: There was an unexpected network error while writing to 192.168.201.43: rc = -22 [ 1605.364345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1605.529892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1605.536390] LNet: 1 local NIs in recovery (showing 1): 192.168.201.43@tcp [ 1605.721718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1613.460537] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:22:26 (1782609746) [ 1613.577647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.624556] LNet: 75406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1613.630882] LNetError: 75406:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1613.636875] LNet: Removed LNI 192.168.201.43@tcp [ 1614.713290] LNet: Removed LNI 192.168.201.43@tcp1 [ 1614.776667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.914300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1615.221207] LNet: 75449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1615.782585] Key type .llcrypt unregistered [ 1615.789535] Key type ._llcrypt unregistered [ 1628.569557] Key type ._llcrypt registered [ 1628.573418] Key type .llcrypt registered [ 1630.434281] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:22:43 (1782609763) [ 1630.615807] LNet: 76217:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1630.615807] mask usage: [+|-] ... [ 1630.874753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1630.923050] alg: No test for adler32 (adler32-zlib) [ 1631.896933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.480070] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1635.621246] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:22:49 (1782609769) [ 1635.709997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.743538] LNet: 76540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1635.782323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.855861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1635.878392] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1636.586837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1636.610421] LNet: Added LNI 192.168.201.43@tcp1 [8/256/0/180] [ 1637.475899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 1637.510765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp1 [ 1637.541946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp1 [ 1644.100708] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 21:22:57 (1782609777) [ 1646.213162] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg143-server [ 1647.851830] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 21:23:01 (1782609781) [ 1649.834643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg143-server [ 1651.104615] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 21:23:05 (1782609785) [ 1652.931967] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg143-server [ 1655.127145] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 21:23:08 (1782609788) [ 1657.225085] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg143-server [ 1658.844842] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 21:23:12 (1782609792) [ 1661.026493] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg143-server [ 1663.528296] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 21:23:16 (1782609796) [ 1665.599631] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg143-server [ 1667.947834] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 21:23:21 (1782609801) [ 1670.246936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg143-server [ 1672.202465] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 21:23:26 (1782609806) [ 1674.831106] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg143-server [ 1676.904836] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 21:23:30 (1782609810) [ 1679.029994] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg143-server [ 1680.820848] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 21:23:34 (1782609814) [ 1681.005515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1681.043935] LNet: 80418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1681.054529] LNetError: 80418:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1681.061628] LNet: Removed LNI 192.168.201.43@tcp [ 1682.087458] LNet: Removed LNI 192.168.201.43@tcp1 [ 1682.154535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.264586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1682.300854] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1683.617620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1683.718647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 1683.885983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.920116] LNet: 80572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.936444] LNetError: 80572:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.943035] LNetError: 80572:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1684.967196] LNet: Removed LNI 192.168.201.43@tcp [ 1685.030964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.105898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1685.147162] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1686.130796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 1686.316427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.347988] LNet: 80717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.362983] LNetError: 80717:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.407822] LNet: Removed LNI 192.168.201.43@tcp [ 1687.463839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.533689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1687.556934] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1688.364384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1688.430733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 1690.820465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.839502] LNet: 80883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.845356] LNetError: 80883:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.860982] LNet: Removed LNI 192.168.201.43@tcp [ 1690.880618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.915622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1690.939495] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1691.887186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 1692.196267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1693.292372] LNet: Removed LNI 192.168.201.43@tcp [ 1693.323860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.417475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1693.443966] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1694.448810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1694.537343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 1694.855960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.889936] LNetError: 81176:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.900394] LNetError: 81176:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1694.916974] LNet: Removed LNI 192.168.201.43@tcp [ 1694.944933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.022365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1695.051400] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1695.976732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 1696.159729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1697.191952] LNet: Removed LNI 192.168.201.43@tcp [ 1697.235964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.359274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1697.393168] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1698.591180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1698.805073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 1699.060699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.096721] LNet: 81468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.103365] LNet: 81468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1700.134740] LNet: Removed LNI 192.168.201.43@tcp [ 1700.149505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.257413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1700.328916] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1701.748663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 1702.074371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.157561] LNet: Removed LNI 192.168.201.43@tcp [ 1703.278983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.456817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1703.530637] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1704.474904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1704.580325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 1704.807715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.840956] LNetError: 81761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.851408] LNetError: 81761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1704.869118] LNet: Removed LNI 192.168.201.43@tcp [ 1704.965961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.204829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1705.258610] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1706.512532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 1706.797611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.881590] LNet: Removed LNI 192.168.201.43@tcp [ 1707.927909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.023529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1708.067302] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1709.296851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1709.463809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 1709.817343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.952349] LNet: Removed LNI 192.168.201.43@tcp [ 1711.033418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.143589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1711.200571] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1712.609584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 1714.192066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.238376] LNet: Removed LNI 192.168.201.43@tcp [ 1715.279550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.337494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1715.376282] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1716.039575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1716.099967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.121507] LNet: 82341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.124297] LNet: 82341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1716.138290] LNet: Removed LNI 192.168.201.43@tcp [ 1716.146672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.178567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1716.188215] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1716.935757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1725.075919] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 21:24:19 (1782609859) [ 1725.352283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.401841] LNetError: 83045:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1725.422707] LNetError: 83045:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1725.447284] LNet: Removed LNI 192.168.201.43@tcp [ 1725.474147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.531530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1725.547842] LNet: Added LNI 192.168.201.43@tcp231 [8/256/0/180] [ 1726.746476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.776292] LNet: Removed LNI 192.168.201.43@tcp231 [ 1726.801524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.922463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1727.003176] LNet: Added LNI 192.168.201.43@tcp231 [8/256/0/180] [ 1727.018764] LNet: 83191:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1727.749434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1727.792537] LNet: Removed LNI 192.168.201.43@tcp231 [ 1727.824637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1727.865567] LNet: Added LNI 192.168.201.43@tcp231 [8/256/0/180] [ 1729.137945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.173443] LNet: Removed LNI 192.168.201.43@tcp231 [ 1729.196422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.297512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1729.333322] LNet: Added LNI 192.168.201.43@tcp231 [8/256/0/0] [ 1729.340778] LNet: 83344:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1729.672776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1729.694221] LNet: Removed LNI 192.168.201.43@tcp231 [ 1729.710488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1729.726652] LNet: Added LNI 192.168.201.43@tcp231 [8/256/0/0] [ 1730.553803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.583243] LNet: Removed LNI 192.168.201.43@tcp231 [ 1730.621074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.721667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1730.773961] LNet: Added LNI 192.168.201.43@tcp231 [8/256/0/60] [ 1730.785912] LNet: 83495:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1731.209380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1731.238946] LNet: Removed LNI 192.168.201.43@tcp231 [ 1731.275607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1731.310691] LNet: Added LNI 192.168.201.43@tcp231 [8/256/0/60] [ 1732.436858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1732.461713] LNet: Removed LNI 192.168.201.43@tcp231 [ 1732.490279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.557413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1732.586543] LNet: Added LNI 192.168.201.43@tcp231 [8/256/0/180] [ 1733.162221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1733.203693] LNet: Removed LNI 192.168.201.43@tcp231 [ 1733.223859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1733.266141] LNet: Added LNI 192.168.201.43@tcp231 [8/256/0/180] [ 1742.858452] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 21:24:36 (1782609876) [ 1742.975982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.047357] LNet: Removed LNI 192.168.201.43@tcp231 [ 1743.100356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.162421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1743.185798] LNet: Added LNI 192.168.201.43@tcp232 [8/256/0/180] [ 1743.988724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1744.027867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1744.068179] LNet: Removed LNI 192.168.201.43@tcp232 [ 1744.101539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1744.152562] LNet: Added LNI 192.168.201.43@tcp232 [8/256/0/180] [ 1744.795709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1744.825145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1744.849364] LNet: Removed LNI 192.168.201.43@tcp232 [ 1744.870320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1744.890578] LNet: Added LNI 192.168.201.43@tcp232 [8/256/0/180] [ 1745.504449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1745.552235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1745.590543] LNet: Removed LNI 192.168.201.43@tcp232 [ 1745.615952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1745.666270] LNet: Added LNI 192.168.201.43@tcp232 [8/256/0/180] [ 1746.388838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1746.418323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1746.444700] LNet: Removed LNI 192.168.201.43@tcp232 [ 1746.472210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1752.444725] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 21:24:46 (1782609886) [ 1753.077880] LNet: 85484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.086648] LNet: 85484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1753.667380] Key type .llcrypt unregistered [ 1753.671257] Key type ._llcrypt unregistered [ 1760.245723] Key type ._llcrypt registered [ 1760.251333] Key type .llcrypt registered [ 1760.299440] LNet: 85713:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1760.299440] mask usage: [+|-] ... [ 1760.518777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1760.527697] alg: No test for adler32 (adler32-zlib) [ 1761.475272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1761.511762] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1769.043784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.143@tcp [ 1778.730038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.800865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1778.867975] LNet: Added LNI 192.168.201.43@tcp1 [8/256/0/180] [ 1781.463750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_interrupt -r 2 [ 1781.491784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp1 -m GET -e local_interrupt -r 2 [ 1781.517383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_interrupt -r 2 [ 1781.549847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -m GET -e local_interrupt -r 2 [ 1781.579551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp -m GET -e local_interrupt -r 2 [ 1781.605872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp1 -m GET -e local_interrupt -r 2 [ 1781.637051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -m GET -e local_interrupt -r 2 [ 1781.672356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -m GET -e local_interrupt -r 2 [ 1781.705336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.43@tcp 192.168.201.143@tcp [ 1781.725732] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1781.729883] LNet: 1 local NIs in recovery (showing 1): 192.168.201.43@tcp [ 1782.080152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1782.112796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1782.179661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1782.212263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1782.275331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1782.314584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1782.347962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1782.379466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1782.394679] LNet: 86147:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1782.408883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1782.447455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1782.468297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1782.490769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1782.524163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1782.545512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1782.972157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_dropped -r 2 [ 1783.030352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp1 -m GET -e local_dropped -r 2 [ 1783.075697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_dropped -r 2 [ 1783.122908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -m GET -e local_dropped -r 2 [ 1783.195450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp -m GET -e local_dropped -r 2 [ 1783.234686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp1 -m GET -e local_dropped -r 2 [ 1783.298968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -m GET -e local_dropped -r 2 [ 1783.361530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -m GET -e local_dropped -r 2 [ 1783.448638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.43@tcp 192.168.201.143@tcp [ 1783.491942] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1783.816411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1783.857524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1783.923743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1783.981780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1784.031262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1784.085777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1784.132725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1784.196376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1784.244614] LNet: 86267:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1784.265894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1784.312628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1784.343375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1784.411186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1784.512564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1784.554459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1785.213949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_aborted -r 2 [ 1785.246468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp1 -m GET -e local_aborted -r 2 [ 1785.278502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_aborted -r 2 [ 1785.310572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -m GET -e local_aborted -r 2 [ 1785.342393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp -m GET -e local_aborted -r 2 [ 1785.384473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp1 -m GET -e local_aborted -r 2 [ 1785.418717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -m GET -e local_aborted -r 2 [ 1785.470446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -m GET -e local_aborted -r 2 [ 1785.518191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.43@tcp 192.168.201.143@tcp [ 1785.556484] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1785.876688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1785.940062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1785.996387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1786.063704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1786.146964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1786.205828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1786.234920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1786.268540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1786.284510] LNet: 86387:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1786.310465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1786.341562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1786.377365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1786.409963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1786.440930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1786.478802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1787.225790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_no_route -r 2 [ 1787.260750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp1 -m GET -e local_no_route -r 2 [ 1787.292372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_no_route -r 2 [ 1787.326693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -m GET -e local_no_route -r 2 [ 1787.369584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp -m GET -e local_no_route -r 2 [ 1787.419464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp1 -m GET -e local_no_route -r 2 [ 1787.479436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -m GET -e local_no_route -r 2 [ 1787.513532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -m GET -e local_no_route -r 2 [ 1787.566597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.43@tcp 192.168.201.143@tcp [ 1787.596249] LNet: There was an unexpected network error while writing to 192.168.201.143: rc = -22 [ 1787.952224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1788.003377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1788.053384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1788.112397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1788.147777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1788.176752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1788.203563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1788.254622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1788.289236] LNet: 86507:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1788.309719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1788.340664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1788.370617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1788.400604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1788.435994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1788.468312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1789.071745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e local_timeout -r 2 [ 1789.107260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp1 -m GET -e local_timeout -r 2 [ 1789.144565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e local_timeout -r 2 [ 1789.194918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -m GET -e local_timeout -r 2 [ 1789.231349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp -m GET -e local_timeout -r 2 [ 1789.289278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp1 -m GET -e local_timeout -r 2 [ 1789.341325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -m GET -e local_timeout -r 2 [ 1789.375744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -m GET -e local_timeout -r 2 [ 1789.414727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.43@tcp 192.168.201.143@tcp [ 1789.687200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1789.733185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1789.772256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1789.804746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1789.839269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1789.906137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1789.942461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1789.978460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1790.033908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1790.079849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1790.145254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1790.194692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1790.234346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1790.271349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1790.848272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp -m GET -e remote_dropped -r 2 [ 1790.890339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.143@tcp1 -m GET -e remote_dropped -r 2 [ 1790.929585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp -m GET -e remote_dropped -r 2 [ 1790.967235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp -d 192.168.201.43@tcp1 -m GET -e remote_dropped -r 2 [ 1791.007304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp -m GET -e remote_dropped -r 2 [ 1791.049395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.143@tcp1 -m GET -e remote_dropped -r 2 [ 1791.101732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp -m GET -e remote_dropped -r 2 [ 1791.196663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.43@tcp1 -d 192.168.201.43@tcp1 -m GET -e remote_dropped -r 2 [ 1791.273510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.43@tcp 192.168.201.143@tcp [ 1791.309397] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.143@tcp [ 1791.560552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1791.620785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1791.650528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1791.693906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1791.723574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1791.775105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1791.824275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1791.874734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1791.923372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1791.957200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1791.998275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1792.040849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1792.078353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1792.116534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1798.613362] LNet: 86822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.639240] LNetError: 86822:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.650901] LNet: Removed LNI 192.168.201.43@tcp [ 1798.670833] LNet: Removed LNI 192.168.201.43@tcp1 [ 1799.623753] Key type .llcrypt unregistered [ 1799.630819] Key type ._llcrypt unregistered [ 1818.740468] Key type ._llcrypt registered [ 1818.742454] Key type .llcrypt registered [ 1820.122912] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 21:25:53 (1782609953) [ 1820.392496] LNet: 87627:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1820.392496] mask usage: [+|-] ... [ 1820.579381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1820.599866] alg: No test for adler32 (adler32-zlib) [ 1821.551777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.657101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.688118] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1822.852907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1822.903654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.43@tcp [ 1822.935178] LNetError: 87802:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.201.43@tcp to recovery [ 1823.091945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.43@tcp [ 1823.227374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1831.156739] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 21:26:04 (1782609964) [ 1831.299719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1831.327631] LNet: 88387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1831.336921] LNetError: 88387:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.368479] LNet: Removed LNI 192.168.201.43@tcp [ 1831.419912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1831.528392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1831.551653] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1832.351669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1832.387461] LNet: Added LNI 192.168.201.43@tcp1 [8/256/0/180] [ 1833.415633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.43@tcp [ 1833.437840] LNetError: 88659:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.201.43@tcp to recovery [ 1833.504698] LNet: 1 local NIs in recovery (showing 1): 192.168.201.43@tcp [ 1833.507651] LNet: There was an unexpected network error while writing to 192.168.201.43: rc = -22 [ 1834.799447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.43@tcp [ 1842.543959] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 21:26:16 (1782609976) [ 1844.935928] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg143-server [ 1846.629977] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 21:26:20 (1782609980) [ 1848.731507] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg143-server [ 1850.670527] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 21:26:24 (1782609984) [ 1852.749686] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg143-server [ 1854.796300] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 21:26:28 (1782609988) [ 1854.949322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1854.985076] LNet: 90277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1854.991826] LNetError: 90277:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1854.999652] LNet: Removed LNI 192.168.201.43@tcp [ 1855.008576] LNet: Removed LNI 192.168.201.43@tcp1 [ 1855.040987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1855.100283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1855.160360] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1855.900503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1855.972605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1856.015720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1856.103562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1856.144753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1856.185899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1863.475541] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 21:26:37 (1782609997) [ 1863.622565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1863.651916] LNet: 91002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1863.661265] LNetError: 91002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1863.671664] LNetError: 91002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1863.689881] LNet: Removed LNI 192.168.201.43@tcp [ 1863.710437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1863.808744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1863.849236] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1872.399352] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 21:26:46 (1782610006) [ 1872.639854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.678547] LNet: 91711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.693719] LNetError: 91711:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.710106] LNet: Removed LNI 192.168.201.43@tcp [ 1872.723968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.777876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1872.835705] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1881.072393] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 21:26:54 (1782610014) [ 1883.003567] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1884.801797] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 21:26:58 (1782610018) [ 1885.454325] LNet: 92746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.463322] LNetError: 92746:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1885.478182] LNet: Removed LNI 192.168.201.43@tcp [ 1886.474484] Key type .llcrypt unregistered [ 1886.478613] Key type ._llcrypt unregistered [ 1893.702783] Key type ._llcrypt registered [ 1893.704721] Key type .llcrypt registered [ 1893.761662] LNet: 92975:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1893.761662] mask usage: [+|-] ... [ 1893.931408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1893.949305] alg: No test for adler32 (adler32-zlib) [ 1894.931532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1894.954871] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1902.667330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.143@tcp [ 1907.093518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.143@tcp [ 1907.603411] LNet: 93164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1907.610844] LNetError: 93164:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1907.632429] LNet: Removed LNI 192.168.201.43@tcp [ 1908.098725] Key type .llcrypt unregistered [ 1908.106439] Key type ._llcrypt unregistered [ 1922.287609] Key type ._llcrypt registered [ 1922.290310] Key type .llcrypt registered [ 1923.750206] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1925.925208] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1927.895280] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 21:27:41 (1782610061) [ 1929.720777] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1932.136199] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 21:27:45 (1782610065) [ 1934.272699] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg143-server [ 1936.415993] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 21:27:50 (1782610070) [ 1938.744318] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg143-server [ 1940.673538] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 21:27:54 (1782610074) [ 1941.802503] LNet: 95216:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1941.802503] mask usage: [+|-] ... [ 1942.193986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1942.418872] alg: No test for adler32 (adler32-zlib) [ 1943.494120] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1943.717784] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 1945.402980] Key type lgssc registered [ 1946.852514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1969.304565] Key type lgssc unregistered [ 1969.555881] LNet: 96339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.560996] LNetError: 96339:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1969.577826] LNet: Removed LNI 192.168.201.43@tcp [ 1970.270332] Key type .llcrypt unregistered [ 1970.272000] Key type ._llcrypt unregistered [ 1991.418363] Key type ._llcrypt registered [ 1991.420667] Key type .llcrypt registered [ 1992.764188] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 21:28:46 (1782610126) [ 1993.153163] LNet: 97136:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1993.153163] mask usage: [+|-] ... [ 1993.331057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1993.352558] alg: No test for adler32 (adler32-zlib) [ 1994.296822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1994.374090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1994.440171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1994.504769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1994.565910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2002.323679] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 21:28:56 (1782610136) [ 2002.469944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2002.525677] LNet: 97747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2002.568728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.731205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2002.746323] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2003.454840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2003.471473] LNet: 97898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.477281] LNetError: 97898:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2003.488907] LNet: Removed LNI 192.168.201.43@tcp [ 2003.510361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2003.567640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2003.606340] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2004.071065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2004.124147] LNetError: 97926:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2004.146789] LNet: Removed LNI 192.168.201.43@tcp [ 2004.167727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2004.305326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2004.372631] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2014.157888] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 21:29:07 (1782610147) [ 2014.299323] LNet: 98508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2014.305890] LNet: 98508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2014.312796] LNetError: 98508:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2014.340220] LNet: Removed LNI 192.168.201.43@tcp [ 2015.460432] Key type .llcrypt unregistered [ 2015.463293] Key type ._llcrypt unregistered [ 2023.393346] Key type ._llcrypt registered [ 2023.395821] Key type .llcrypt registered [ 2023.444530] LNet: 98749:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2023.444530] mask usage: [+|-] ... [ 2023.525546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2023.538979] alg: No test for adler32 (adler32-zlib) [ 2024.459331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2024.523364] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2024.525970] LNet: *** cfs_fail_loc=e003, val=100*** [ 2025.157408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 2025.174421] LNet: *** cfs_fail_loc=e003, val=100*** [ 2025.177436] LNet: Skipped 2 previous similar messages [ 2025.183904] Ping reply received with mapping: [ 2025.186498] found metadata reply NI[192.168.201.43@tcp] [ 2025.190433] 2b c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2025.196593] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2025.200883] 5a 5a 5a 5a 5a 5a 5a 5a [ 2025.310789] LNet: 98915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2025.315544] LNetError: 98915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2026.369539] LNet: Removed LNI 192.168.201.43@tcp [ 2027.403508] Key type .llcrypt unregistered [ 2027.410749] Key type ._llcrypt unregistered [ 2043.146298] Key type ._llcrypt registered [ 2043.149511] Key type .llcrypt registered [ 2044.511399] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 21:29:38 (1782610178) [ 2045.162677] Key type .llcrypt unregistered [ 2045.164887] Key type ._llcrypt unregistered [ 2052.043556] Key type ._llcrypt registered [ 2052.046480] Key type .llcrypt registered [ 2052.084561] LNet: 99911:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2052.084561] mask usage: [+|-] ... [ 2052.371702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2052.397938] alg: No test for adler32 (adler32-zlib) [ 2053.586721] LNetError: 99961:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2054.005327] Key type .llcrypt unregistered [ 2054.014991] Key type ._llcrypt unregistered [ 2060.036553] Key type ._llcrypt registered [ 2060.038666] Key type .llcrypt registered [ 2061.387155] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 21:29:55 (1782610195) [ 2062.264212] Key type .llcrypt unregistered [ 2062.266568] Key type ._llcrypt unregistered [ 2069.849303] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2069.853117] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2072.555704] Key type ._llcrypt registered [ 2072.558517] Key type .llcrypt registered [ 2072.638768] LNet: 100829:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2072.638768] mask usage: [+|-] ... [ 2072.892545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2072.927214] alg: No test for adler32 (adler32-zlib) [ 2073.973252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2074.088654] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2086.358930] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 21:30:19 (1782610219) [ 2086.520223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2086.573516] LNet: 101735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.593840] LNetError: 101735:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2086.640170] LNet: Removed LNI 10.1.2.3@tcp [ 2086.676552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2087.760375] Key type .llcrypt unregistered [ 2087.766199] Key type ._llcrypt unregistered [ 2095.696186] Key type ._llcrypt registered [ 2095.699184] Key type .llcrypt registered [ 2095.770177] LNet: 101974:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2095.770177] mask usage: [+|-] ... [ 2096.127218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2096.185172] alg: No test for adler32 (adler32-zlib) [ 2097.209733] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2097.351815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 2097.615595] LNet: 102032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2097.625845] LNetError: 102032:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2097.643432] LNet: Removed LNI 192.168.201.43@tcp [ 2098.619544] Key type .llcrypt unregistered [ 2098.625691] Key type ._llcrypt unregistered [ 2113.562441] Key type ._llcrypt registered [ 2113.568622] Key type .llcrypt registered [ 2115.153544] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 21:30:48 (1782610248) [ 2115.889763] Key type .llcrypt unregistered [ 2115.893933] Key type ._llcrypt unregistered [ 2124.812977] Key type ._llcrypt registered [ 2124.818225] Key type .llcrypt registered [ 2124.862974] LNet: 103016:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2124.862974] mask usage: [+|-] ... [ 2125.051154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2125.082965] alg: No test for adler32 (adler32-zlib) [ 2133.321268] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 21:31:07 (1782610267) [ 2133.891876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2133.934763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.024100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2134.070740] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2134.804610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2134.869263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2134.922327] LNet: Removed LNI 192.168.201.43@tcp [ 2135.315609] LNet: 103776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2136.003493] Key type .llcrypt unregistered [ 2136.006484] Key type ._llcrypt unregistered [ 2150.739681] Key type ._llcrypt registered [ 2150.741497] Key type .llcrypt registered [ 2151.917824] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 21:31:26 (1782610286) [ 2152.495118] LNet: 104543:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2152.495118] mask usage: [+|-] ... [ 2152.670353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2152.680551] alg: No test for adler32 (adler32-zlib) [ 2153.737681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.812921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2153.832657] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2154.775640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 2154.817637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.43@tcp [ 2154.838733] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2154.842602] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2154.846867] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2154, 0, 0 [ 2154.852909] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2154.855656] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2154.863030] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2154.867639] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2154, 0, 0 [ 2154.871958] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2154.874803] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2154.878321] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2154.882339] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2154, 0, 2204 [ 2154.888320] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2154.892867] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2154.897718] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2154.904890] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2154, 0, 2204 [ 2154.910869] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2163.250646] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 21:31:36 (1782610296) [ 2164.098817] LNet: 105318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2164.116759] LNetError: 105318:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2164.148470] LNet: Removed LNI 192.168.201.43@tcp [ 2165.086675] Key type .llcrypt unregistered [ 2165.089779] Key type ._llcrypt unregistered [ 2175.120690] Key type ._llcrypt registered [ 2175.126562] Key type .llcrypt registered [ 2175.176757] LNet: 105542:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2175.176757] mask usage: [+|-] ... [ 2175.463378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2175.478814] alg: No test for adler32 (adler32-zlib) [ 2176.571840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2176.609419] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2187.494224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.143@tcp [ 2198.007377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2198.078941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2198.117264] LNet: Added LNI 192.168.201.43@tcp1 [8/256/0/180] [ 2199.599994] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2199.893966] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2199.904458] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2199.912411] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2202.619578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2202.674735] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2215.913954] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 21:32:29 (1782610349) [ 2216.148308] LNet: 106828:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2216.154685] LNetError: 106828:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2216.164965] LNet: Removed LNI 192.168.201.43@tcp [ 2217.185246] LNet: Removed LNI 192.168.201.43@tcp1 [ 2217.193235] LNetError: 106828:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2217.205024] LNetError: 106828:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2217.222817] LNet: Removed LNI 10.1.2.3@tcp99 [ 2218.025623] Key type .llcrypt unregistered [ 2218.030595] Key type ._llcrypt unregistered [ 2225.814392] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2225.825398] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2228.696868] Key type ._llcrypt registered [ 2228.699604] Key type .llcrypt registered [ 2228.779811] LNet: 107094:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2228.779811] mask usage: [+|-] ... [ 2229.081056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2229.119771] alg: No test for adler32 (adler32-zlib) [ 2230.151101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2230.239527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2230.282785] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2231.342823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2231.378144] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2234.126093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2234.211908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2234.305397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2234.343312] LNetError: 107572:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2234.386557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2234.483537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2235.176664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2235.359227] LNet: 107593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2235.385650] LNetError: 107593:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2235.399461] LNet: Removed LNI 10.1.2.3@tcp [ 2236.461183] LNet: Removed LNI 192.168.201.43@tcp [ 2237.199636] Key type .llcrypt unregistered [ 2237.202360] Key type ._llcrypt unregistered [ 2249.923465] Key type ._llcrypt registered [ 2249.925338] Key type .llcrypt registered [ 2251.503914] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 21:33:05 (1782610385) [ 2251.711866] LNet: 108389:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2251.711866] mask usage: [+|-] ... [ 2251.910214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2251.932229] alg: No test for adler32 (adler32-zlib) [ 2252.987950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2253.067177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2253.115880] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2254.144736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.43@tcp [ 2254.254819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg143-client@tcp [ 2260.771336] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 21:33:14 (1782610394) [ 2261.123420] LNet: 109128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2261.146403] LNetError: 109128:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2261.193250] LNet: Removed LNI 192.168.201.43@tcp [ 2261.973190] Key type .llcrypt unregistered [ 2261.976137] Key type ._llcrypt unregistered [ 2270.016680] Key type ._llcrypt registered [ 2270.020281] Key type .llcrypt registered [ 2270.316813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2270.367211] alg: No test for adler32 (adler32-zlib) [ 2271.351819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2271.378396] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2271.430901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2279.550865] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 21:33:32 (1782610412) [ 2281.391801] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2283.387499] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 21:33:37 (1782610417) [ 2285.724283] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2287.676965] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 21:33:41 (1782610421) [ 2287.849126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2287.877657] LNet: 110547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2287.882242] LNetError: 110547:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2287.898817] LNet: Removed LNI 192.168.201.43@tcp [ 2287.926389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2288.016737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2288.064155] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2289.273257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2289.303482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2296.863578] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 21:33:51 (1782610431) [ 2297.258466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2297.302437] LNet: 111256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2297.313624] LNetError: 111256:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2297.328336] LNet: Removed LNI 192.168.201.43@tcp [ 2297.356432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2297.442808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2297.481834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2298.385344] Key type .llcrypt unregistered [ 2298.387245] Key type ._llcrypt unregistered [ 2312.454233] Key type ._llcrypt registered [ 2312.457787] Key type .llcrypt registered [ 2314.196865] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 21:34:07 (1782610447) [ 2314.497567] LNet: 112064:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2314.497567] mask usage: [+|-] ... [ 2314.734165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2314.757931] alg: No test for adler32 (adler32-zlib) [ 2315.686570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2315.740614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2315.781501] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2316.643549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2316.710494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2324.594716] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 21:34:18 (1782610458) [ 2324.707147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2324.744513] LNet: 112804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.752663] LNetError: 112804:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2324.766575] LNet: Removed LNI 192.168.201.43@tcp [ 2324.813900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2324.931751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2324.983357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2332.714752] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 21:34:26 (1782610466) [ 2332.833417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2332.850581] LNet: 113385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2332.878385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2332.977485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2333.654420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2333.729744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2333.811339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2334.467857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2334.538445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2334.686492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2334.722959] LNet: 113426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2334.771576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.878949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2335.422652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2335.477978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2335.549792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2336.221917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2336.303867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2343.976794] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 21:34:37 (1782610477) [ 2344.249038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2344.279983] LNet: 114026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.293746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2344.374357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2344.409390] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2345.526865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2345.554961] LNet: Added LNI 192.168.201.43@tcp2 [8/256/0/180] [ 2346.595454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2346.677352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2346.972691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2346.998857] LNetError: 114306:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2347.006357] LNet: Removed LNI 192.168.201.43@tcp [ 2347.017521] LNet: Removed LNI 192.168.201.43@tcp2 [ 2347.039448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.187663] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2347.205729] LNet: Added LNI 192.168.201.43@tcp2 [8/256/0/180] [ 2347.217132] LNet: 114312:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2347.910716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2348.148886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2348.167906] LNetError: 114349:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2348.173290] LNetError: 114349:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2348.179314] LNet: Removed LNI 192.168.201.43@tcp [ 2348.198706] LNet: Removed LNI 192.168.201.43@tcp2 [ 2348.235644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2348.291303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2348.310191] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2349.390382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2349.438732] LNet: Added LNI 192.168.201.43@tcp2 [8/256/0/180] [ 2350.490043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2350.526556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2350.583267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2350.964187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2351.011405] LNet: 114633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2351.020723] LNet: 114633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2351.033781] LNetError: 114633:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2351.040191] LNetError: 114633:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2351.045530] LNet: Removed LNI 192.168.201.43@tcp [ 2352.119498] LNet: Removed LNI 192.168.201.43@tcp2 [ 2352.187729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2352.283103] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2352.297404] LNet: Added LNI 192.168.201.43@tcp2 [8/256/0/180] [ 2352.302844] LNet: 114639:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2353.374730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2353.814642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2353.871226] LNet: Removed LNI 192.168.201.43@tcp [ 2354.934106] LNet: Removed LNI 192.168.201.43@tcp2 [ 2354.996245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2355.061764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2355.097155] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2355.923599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2355.949863] LNet: Added LNI 192.168.201.43@tcp2 [8/256/0/180] [ 2357.050175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2357.122769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2357.411520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2357.444648] LNetError: 114956:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2357.451661] LNetError: 114956:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2357.458882] LNet: Removed LNI 192.168.201.43@tcp [ 2357.488985] LNet: Removed LNI 192.168.201.43@tcp2 [ 2357.537780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2357.623790] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2357.642781] LNet: Added LNI 192.168.201.43@tcp2 [8/256/0/180] [ 2357.647582] LNet: 114962:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2358.530542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2358.817598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2359.905833] LNet: Removed LNI 192.168.201.43@tcp [ 2359.922718] LNet: Removed LNI 192.168.201.43@tcp2 [ 2359.965294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.064302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2360.098357] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2361.192095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2361.221949] LNet: Added LNI 192.168.201.43@tcp2 [8/256/0/180] [ 2362.369675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2362.420449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2362.473718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2362.740191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2362.762485] LNet: 115284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2362.767217] LNet: 115284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2363.808392] LNet: Removed LNI 192.168.201.43@tcp [ 2364.841082] LNet: Removed LNI 192.168.201.43@tcp2 [ 2364.889494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2364.999638] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2365.010634] LNet: Added LNI 192.168.201.43@tcp2 [8/256/0/180] [ 2365.023106] LNet: 115290:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2365.430294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2365.768669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.790401] LNetError: 115327:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2365.796699] LNetError: 115327:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2365.802863] LNet: Removed LNI 192.168.201.43@tcp [ 2366.822776] LNet: Removed LNI 192.168.201.43@tcp2 [ 2366.868503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2366.943953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2366.968351] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2368.038971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2368.088972] LNet: Added LNI 192.168.201.43@tcp2 [8/256/0/180] [ 2369.137063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2369.183374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2369.305134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2369.786913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2370.848276] LNet: Removed LNI 192.168.201.43@tcp [ 2370.860465] LNet: Removed LNI 192.168.201.43@tcp2 [ 2370.895141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2370.964858] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2370.975647] LNet: Added LNI 192.168.201.43@tcp2 [8/256/0/180] [ 2370.987483] LNet: 115630:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2371.704461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2372.251164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2373.344238] LNet: Removed LNI 192.168.201.43@tcp [ 2373.357468] LNet: Removed LNI 192.168.201.43@tcp2 [ 2373.430305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2373.490520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2373.514902] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2374.176928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2374.203687] LNet: Added LNI 192.168.201.43@tcp2 [8/256/0/180] [ 2375.131197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2375.172358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2375.205239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2375.267359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2375.575864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2376.608464] LNet: Removed LNI 192.168.201.43@tcp [ 2377.641237] LNet: Removed LNI 192.168.201.43@tcp2 [ 2377.673527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2377.795709] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2377.811809] LNet: Added LNI 192.168.201.43@tcp2 [8/256/0/180] [ 2378.687810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2387.203189] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2389.335344] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 21:35:22 (1782610522) [ 2389.513922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2389.546711] LNet: 116745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2389.557846] LNet: 116745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2389.581958] LNetError: 116745:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2389.590739] LNetError: 116745:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2389.598783] LNet: Removed LNI 192.168.201.43@tcp [ 2390.630978] LNet: Removed LNI 192.168.201.43@tcp2 [ 2390.697214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2391.058112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2391.079559] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2391.884399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2391.904124] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2392.909172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2392.924978] LNet: Removed LNI 10.1.2.3@tcp [ 2392.935048] LNet: Removed LNI 192.168.201.43@tcp [ 2392.952717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2393.007604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2393.032884] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2393.037897] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2393.500669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2393.522892] LNet: Removed LNI 192.168.201.43@tcp [ 2393.532924] LNet: Removed LNI 10.1.2.3@tcp [ 2401.440952] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 21:35:35 (1782610535) [ 2401.732780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2401.809286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2401.942491] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2402.117857] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2402.126763] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2402.133183] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2404.516182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2404.534145] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2405.531951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2405.551962] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2408.146697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.43@tcp [ 2408.215634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2408.291447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.43@tcp -m PUT -e local_timeout -r 1 [ 2408.428464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2408.553637] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2408.564066] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2408.925483] LNetError: 117687:0:(lib-move.c:4714:lnet_parse()) 192.168.201.43@tcp, src 192.168.201.43@tcp: Dropping PUT (error -108 looking up sender) [ 2410.977029] LNet: Removed LNI 10.1.2.3@tcp [ 2412.028441] LNet: Removed LNI 192.168.201.43@tcp [ 2412.412847] Key type .llcrypt unregistered [ 2412.417962] Key type ._llcrypt unregistered [ 2426.299488] Key type ._llcrypt registered [ 2426.307099] Key type .llcrypt registered [ 2428.057265] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 21:36:01 (1782610561) [ 2428.346872] LNet: 118921:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2428.346872] mask usage: [+|-] ... [ 2428.559332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2428.594974] alg: No test for adler32 (adler32-zlib) [ 2429.612553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2430.197805] LNet: 119002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2430.815086] Key type .llcrypt unregistered [ 2430.821513] Key type ._llcrypt unregistered [ 2438.277985] Key type ._llcrypt registered [ 2438.280342] Key type .llcrypt registered [ 2438.327516] LNet: 119231:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2438.327516] mask usage: [+|-] ... [ 2438.491285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2438.504292] alg: No test for adler32 (adler32-zlib) [ 2439.502016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2439.550969] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2448.245886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.143@tcp [ 2448.557339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2449.614566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.143@tcp [ 2449.827320] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.143@tcp [ 2450.006905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2450.066801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2450.132371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2450.167675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2450.204563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2450.245811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2450.284349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2450.335785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2450.367400] LNet: 119452:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2450.392739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2450.456792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2450.510557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2450.574092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2450.627400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2450.667371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2451.483652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2451.813274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2451.859349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2451.891763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2451.922685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2451.961837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2451.995803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2452.037376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2452.074580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2452.113595] LNet: 119569:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2452.160753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2452.247790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2452.332926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2452.401389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2452.482247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2452.516889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2453.794754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.143@tcp [ 2454.268870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2454.326675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2454.374840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2454.447899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2454.491830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2454.526934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2454.561762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2454.590854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2454.612586] LNet: 119687:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2454.629760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2454.659311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2454.687527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2454.758754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2454.824942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2454.872229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2455.638764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2455.869627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2455.897701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2455.934677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2455.987921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2456.042596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2456.094854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2456.121463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2456.164343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2456.212549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2456.252313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2456.322539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2456.381850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2456.424247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2456.484795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2456.800319] LNet: 119850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2456.818685] LNetError: 119850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2456.849375] LNet: Removed LNI 192.168.201.43@tcp [ 2457.405816] Key type .llcrypt unregistered [ 2457.410338] Key type ._llcrypt unregistered [ 2476.986251] Key type ._llcrypt registered [ 2476.989294] Key type .llcrypt registered [ 2478.597500] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 21:36:52 (1782610612) [ 2478.828247] LNet: 120655:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2478.828247] mask usage: [+|-] ... [ 2479.042458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2479.070155] alg: No test for adler32 (adler32-zlib) [ 2480.090439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2480.583740] LNet: 120738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2481.243457] Key type .llcrypt unregistered [ 2481.247433] Key type ._llcrypt unregistered [ 2488.381092] Key type ._llcrypt registered [ 2488.383302] Key type .llcrypt registered [ 2488.446686] LNet: 120967:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2488.446686] mask usage: [+|-] ... [ 2488.640606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2488.657841] alg: No test for adler32 (adler32-zlib) [ 2489.652625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2489.686970] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2497.077942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.143@tcp [ 2506.186348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2506.252416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2506.316558] LNet: Added LNI 192.168.201.43@tcp1 [8/256/0/180] [ 2507.938802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2508.812897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.143@tcp [ 2509.102384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2509.143582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2509.152736] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.143@tcp [ 2509.225893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2509.280485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2509.365687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2509.401972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2509.451560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2509.513179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2509.546143] LNet: 121392:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2509.591813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2509.636907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2509.699514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2509.765984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2509.824878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2509.885545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2510.976452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2511.293755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2511.340982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2511.366460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2511.398906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2511.436506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2511.466660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2511.504637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2511.554419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2511.588724] LNet: 121509:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2511.619811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2511.676648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2511.725055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2511.784685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2511.815657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2511.858740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2512.638831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.143@tcp [ 2512.890856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2512.919654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2512.962109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2512.993767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2513.034672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2513.075386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2513.117486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2513.156236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2513.175711] LNet: 121626:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2513.213807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2513.264280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2513.324277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2513.380918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2513.455431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2513.519379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2514.320672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2514.618244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2514.654963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2514.691428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2514.729311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2514.759844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2514.789966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2514.822851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2514.860429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2514.895942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2514.949436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2514.990596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2515.039784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2515.095131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2515.142388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2520.445905] LNet: 121825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2520.461979] LNetError: 121825:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2520.473804] LNet: Removed LNI 192.168.201.43@tcp [ 2520.499561] LNet: Removed LNI 192.168.201.43@tcp1 [ 2520.995342] Key type .llcrypt unregistered [ 2520.997516] Key type ._llcrypt unregistered [ 2540.374527] Key type ._llcrypt registered [ 2540.376930] Key type .llcrypt registered [ 2541.829962] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 21:37:55 (1782610675) [ 2550.172059] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 21:38:03 (1782610683) [ 2559.043420] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 21:38:12 (1782610692) [ 2567.185552] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 21:38:21 (1782610701) [ 2576.493139] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 21:38:30 (1782610710) [ 2584.477242] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 21:38:38 (1782610718) [ 2592.562217] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 21:38:46 (1782610726) [ 2600.929244] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 21:38:54 (1782610734) [ 2609.143770] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 21:39:03 (1782610743) [ 2617.818841] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 21:39:11 (1782610751) [ 2627.302097] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 21:39:20 (1782610760) [ 2637.857680] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 21:39:31 (1782610771) [ 2645.163991] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 21:39:38 (1782610778) [ 2653.486859] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 21:39:46 (1782610786) [ 2662.827300] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 21:39:56 (1782610796) [ 2670.569826] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 21:40:04 (1782610804) [ 2679.577650] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 21:40:13 (1782610813) [ 2687.950120] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 21:40:21 (1782610821) [ 2688.307037] LNet: 132370:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2688.307037] mask usage: [+|-] ... [ 2688.462292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2688.477973] alg: No test for adler32 (adler32-zlib) [ 2689.499938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2689.754485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2689.788128] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2691.610236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2691.636502] LNet: 132541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2691.644256] LNetError: 132541:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2691.658751] LNet: Removed LNI 192.168.201.43@tcp [ 2691.680580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2691.748568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2691.810358] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2700.974604] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 21:40:34 (1782610834) [ 2711.413987] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 21:40:44 (1782610844) [ 2722.304106] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 21:40:55 (1782610855) [ 2733.374407] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 21:41:06 (1782610866) [ 2742.495365] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 21:41:15 (1782610875) [ 2752.467222] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 21:41:26 (1782610886) [ 2761.522087] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 21:41:35 (1782610895) [ 2770.660457] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 21:41:44 (1782610904) [ 2778.924867] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 21:41:52 (1782610912) [ 2788.827804] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 21:42:02 (1782610922) [ 2799.843672] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 21:42:12 (1782610932) [ 2809.903385] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 21:42:23 (1782610943) [ 2819.616814] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2557 sec ========== 21:42:33 (1782610953) [ 2820.173965] LNet: 140010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2820.183581] LNetError: 140010:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2820.206305] LNet: Removed LNI 192.168.201.43@tcp [ 2821.652388] Key type .llcrypt unregistered [ 2821.656426] Key type ._llcrypt unregistered [ 2837.841410] Key type ._llcrypt registered [ 2837.846491] Key type .llcrypt registered [ 2837.969158] LNet: 140467:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2837.969158] mask usage: [+|-] ... [ 2838.414122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2838.428198] alg: No test for adler32 (adler32-zlib) [ 2839.707592] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2840.051080] LNet: Added LNI 192.168.201.43@tcp [8/256/0/180] [ 2841.865241] Key type lgssc registered [ 2844.102855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2916.545569] Lustre: Mounted lustre-client [ 2916.745721] LNet: 141963:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2916.745721] mask usage: [+|-] ... [ 2922.441542] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2931.975275] Lustre: Unmounted lustre-client [ 2958.579599] Key type lgssc unregistered [ 2958.889624] LNet: 142947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2958.909800] LNetError: 142947:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2959.978235] LNet: Removed LNI 192.168.201.43@tcp [ 2961.051272] Key type .llcrypt unregistered [ 2961.053909] Key type ._llcrypt unregistered