[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 476850894 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002352] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007344] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.011013] LSM: Security Framework initializing [ 0.012063] Yama: becoming mindful. [ 0.013044] SELinux: Initializing. [ 0.014069] *** VALIDATE selinux *** [ 0.022444] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027775] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030124] *** VALIDATE tmpfs *** [ 0.031486] *** VALIDATE proc *** [ 0.033148] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035319] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038025] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042453] debug: unmapping init [mem 0xffffffff8b459000-0xffffffff8b460fff] [ 0.044178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045746] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046032] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049008] ... value mask: 0000ffffffffffff [ 0.050009] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053345] rcu: Hierarchical SRCU implementation. [ 0.055669] smp: Bringing up secondary CPUs ... [ 0.056678] x86: Booting SMP configuration: [ 0.057024] .... node #0, CPUs: #1 #2 #3 [ 0.060449] smp: Brought up 1 node, 4 CPUs [ 0.062017] smpboot: Max logical packages: 1 [ 0.063025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147397] node 0 deferred pages initialised in 83ms [ 0.152239] devtmpfs: initialized [ 0.154332] x86/mm: Memory block size: 128MB [ 0.158179] gcov: version magic: 0x41383552 [ 0.161514] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165123] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.167294] pinctrl core: initialized pinctrl subsystem [ 0.170199] [ 0.170797] ************************************************************* [ 0.173016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176011] ** ** [ 0.178010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.180015] ** ** [ 0.183013] ** This means that this kernel is built to expose internal ** [ 0.186013] ** IOMMU data structures, which may compromise security on ** [ 0.188011] ** your system. ** [ 0.190016] ** ** [ 0.193014] ** If you see this message and you are not debugging the ** [ 0.195012] ** kernel, report this immediately to your vendor! ** [ 0.198014] ** ** [ 0.200012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.203011] ************************************************************* [ 0.204708] NET: Registered protocol family 16 [ 0.206420] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.209065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.211069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.214094] cpuidle: using governor menu [ 0.215714] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.217480] PCI: Using configuration type 1 for base access [ 0.219196] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.228180] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.229023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.230187] cryptd: max_cpu_qlen set to 1000 [ 0.232236] ACPI: Added _OSI(Module Device) [ 0.234012] ACPI: Added _OSI(Processor Device) [ 0.235011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.240232] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.245337] ACPI: Interpreter enabled [ 0.247058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.248012] ACPI: Using IOAPIC for interrupt routing [ 0.250093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252391] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.262592] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.264036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.267020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.269071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.273368] acpiphp: Slot [2] registered [ 0.275080] acpiphp: Slot [5] registered [ 0.276127] acpiphp: Slot [6] registered [ 0.277085] acpiphp: Slot [3] registered [ 0.279170] acpiphp: Slot [4] registered [ 0.280105] acpiphp: Slot [7] registered [ 0.282095] acpiphp: Slot [8] registered [ 0.283000] acpiphp: Slot [9] registered [ 0.283000] acpiphp: Slot [10] registered [ 0.283000] acpiphp: Slot [11] registered [ 0.285138] acpiphp: Slot [12] registered [ 0.286035] acpiphp: Slot [13] registered [ 0.286975] acpiphp: Slot [14] registered [ 0.288052] acpiphp: Slot [15] registered [ 0.289017] acpiphp: Slot [16] registered [ 0.289960] acpiphp: Slot [17] registered [ 0.291053] acpiphp: Slot [18] registered [ 0.292056] acpiphp: Slot [19] registered [ 0.293089] acpiphp: Slot [20] registered [ 0.295088] acpiphp: Slot [21] registered [ 0.296000] acpiphp: Slot [22] registered [ 0.296000] acpiphp: Slot [23] registered [ 0.297026] acpiphp: Slot [24] registered [ 0.297978] acpiphp: Slot [25] registered [ 0.298055] acpiphp: Slot [26] registered [ 0.299005] acpiphp: Slot [27] registered [ 0.300048] acpiphp: Slot [28] registered [ 0.301114] acpiphp: Slot [29] registered [ 0.303123] acpiphp: Slot [30] registered [ 0.304287] acpiphp: Slot [31] registered [ 0.305140] PCI host bridge to bus 0000:00 [ 0.306015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.308017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.310024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.312031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.314034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.316016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.317186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.320069] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.323269] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.330504] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.334124] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.337037] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.339023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.341026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.343816] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.348010] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.350052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.353952] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.358017] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.369018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.373019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.379839] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.396038] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.408024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.431022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.446711] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.458017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.465019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.484028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.496374] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.498360] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.500331] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.502469] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.504238] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.509253] iommu: Default domain type: Passthrough [ 0.510318] SCSI subsystem initialized [ 0.511105] ACPI: bus type USB registered [ 0.512062] usbcore: registered new interface driver usbfs [ 0.513064] usbcore: registered new interface driver hub [ 0.515080] usbcore: registered new device driver usb [ 0.516130] pps_core: LinuxPPS API ver. 1 registered [ 0.517008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.519080] PTP clock support registered [ 0.522086] EDAC MC: Ver: 3.0.0 [ 0.523114] PCI: Using ACPI for IRQ routing [ 0.524207] NetLabel: Initializing [ 0.524946] NetLabel: domain hash size = 128 [ 0.526010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.527071] NetLabel: unlabeled traffic allowed by default [ 0.529122] vgaarb: loaded [ 0.530208] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.531010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.537000] clocksource: Switched to clocksource kvm-clock [ 0.622569] VFS: Disk quotas dquot_6.6.0 [ 0.624188] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.626617] *** VALIDATE ramfs *** [ 0.627802] *** VALIDATE hugetlbfs *** [ 0.629365] pnp: PnP ACPI init [ 0.631202] pnp: PnP ACPI: found 6 devices [ 0.645259] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.647414] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.649750] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.651751] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.653762] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.655361] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.657511] NET: Registered protocol family 2 [ 0.659639] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.663118] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.665432] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.669645] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.672848] TCP: Hash tables configured (established 65536 bind 65536) [ 0.675650] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.678568] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.681919] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.684494] NET: Registered protocol family 1 [ 0.687432] RPC: Registered named UNIX socket transport module. [ 0.689501] RPC: Registered udp transport module. [ 0.690845] RPC: Registered tcp transport module. [ 0.692562] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.694882] NET: Registered protocol family 44 [ 0.696471] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.698330] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.700127] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.702344] PCI: CLS 0 bytes, default 64 [ 0.704855] Unpacking initramfs... [ 2.101678] debug: unmapping init [mem 0xffff90263cc64000-0xffff90263ffcffff] [ 2.105323] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.107505] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.109923] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.633827] Initialise system trusted keyrings [ 2.635787] Key type blacklist registered [ 2.637875] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.647665] zbud: loaded [ 2.650969] *** VALIDATE nfs *** [ 2.652363] *** VALIDATE nfs4 *** [ 2.656122] pstore: using deflate compression [ 2.659277] Platform Keyring initialized [ 2.773436] NET: Registered protocol family 38 [ 2.774952] Key type asymmetric registered [ 2.776265] Asymmetric key parser 'x509' registered [ 2.777612] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.780708] io scheduler mq-deadline registered [ 2.782188] io scheduler kyber registered [ 2.783306] io scheduler bfq registered [ 2.784665] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.787294] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.789397] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.791715] ACPI: Power Button [PWRF] [ 2.796449] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.802351] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.815237] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.858189] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.887881] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.894461] Non-volatile memory driver v1.3 [ 2.896257] Linux agpgart interface v0.103 [ 2.932406] virtio_blk virtio1: [vda] 149376 512-byte logical blocks (76.5 MB/72.9 MiB) [ 2.935571] vda: detected capacity change from 0 to 76480512 [ 2.955800] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.959734] vdb: detected capacity change from 0 to 1073741824 [ 2.970170] libphy: Fixed MDIO Bus: probed [ 2.975451] usbcore: registered new interface driver usbserial_generic [ 2.978139] usbserial: USB Serial support registered for generic [ 2.980584] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.984699] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.985806] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.987294] mousedev: PS/2 mouse device common for all mice [ 2.990065] rtc_cmos 00:05: RTC can wake from S4 [ 2.992303] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.993473] rtc_cmos 00:05: registered as rtc0 [ 2.996925] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.999547] intel_pstate: CPU model not supported [ 3.002585] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.006705] hid: raw HID events driver (C) Jiri Kosina [ 3.009340] usbcore: registered new interface driver usbhid [ 3.011386] usbhid: USB HID core driver [ 3.011550] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.012549] drop_monitor: Initializing network drop monitor service [ 3.018226] Initializing XFRM netlink socket [ 3.020507] NET: Registered protocol family 10 [ 3.022808] Segment Routing with IPv6 [ 3.024246] NET: Registered protocol family 17 [ 3.026117] mpls_gso: MPLS GSO support [ 3.032098] RAS: Correctable Errors collector initialized. [ 3.034265] AVX version of gcm_enc/dec engaged. [ 3.035742] AES CTR mode by8 optimization enabled [ 3.098162] sched_clock: Marking stable (3098144850, 0)->(3953750420, -855605570) [ 3.100664] registered taskstats version 1 [ 3.103223] Loading compiled-in X.509 certificates [ 3.105191] zswap: loaded using pool lzo/zbud [ 3.126071] Key type big_key registered [ 3.136205] Key type encrypted registered [ 3.137224] ima: No TPM chip found, activating TPM-bypass! [ 3.138321] ima: Allocated hash algorithm: sha1 [ 3.139218] ima: No architecture policies found [ 3.140053] evm: Initialising EVM extended attributes: [ 3.141147] evm: security.selinux [ 3.141880] evm: security.ima [ 3.142564] evm: security.capability [ 3.143340] evm: HMAC attrs: 0x1 [ 3.144590] rtc_cmos 00:05: setting system clock to 2026-08-10 21:48:50 UTC (1786398530) [ 3.148617] debug: unmapping init [mem 0xffffffff8c403000-0xffffffff8c5fffff] [ 3.151200] debug: unmapping init [mem 0xffffffff8b182000-0xffffffff8b458fff] [ 3.160074] Write protecting the kernel read-only data: 28672k [ 3.162856] debug: unmapping init [mem 0xffffffff89803000-0xffffffff899fffff] [ 3.164479] debug: unmapping init [mem 0xffffffff8a114000-0xffffffff8a1fffff] [ 3.188239] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.196764] systemd[1]: Detected virtualization kvm. [ 3.198679] systemd[1]: Detected architecture x86-64. [ 3.200660] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.228129] systemd[1]: No hostname configured. [ 3.229203] systemd[1]: Set hostname to . [ 3.230419] random: systemd: uninitialized urandom read (16 bytes read) [ 3.232180] systemd[1]: Initializing machine ID from random generator. [ 3.347110] random: systemd: uninitialized urandom read (16 bytes read) [ 3.348920] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.352161] random: systemd: uninitialized urandom read (16 bytes read) [ 3.353598] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.356534] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.914435] device-mapper: uevent: version 1.0.3 [ 3.916480] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.712885] virtio_net virtio0 ens2: renamed from eth0 [ 4.881242] scsi host0: ata_piix [ 4.911404] scsi host1: ata_piix [ 4.913149] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.915685] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.280467] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.647512] random: crng init done [ 9.649490] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.566224] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.695443] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.910807] SELinux: Disabled at runtime. [ 11.974818] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.984695] systemd[1]: Detected virtualization kvm. [ 11.986617] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.508681] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.511902] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.516595] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.520696] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.524012] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.530934] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.533745] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ 12.587366] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.906138] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.161877] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.256703] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.309839] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.327520] EDAC sbridge: Ver: 1.1.2 [ 14.297839] Key type dns_resolver registered [ 14.612464] NFS: Registering the id_resolver key type [ 14.614515] Key type id_resolver registered [ 14.616113] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg228-client login: [ 63.370845] libcfs: loading out-of-tree module taints kernel. [ 63.460155] Key type ._llcrypt registered [ 63.475127] Key type .llcrypt registered [ 64.425309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 64.438799] alg: No test for adler32 (adler32-zlib) [ 65.708865] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 66.403717] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 68.287157] Key type lgssc registered [ 69.482699] hrtimer: interrupt took 4689046 ns [ 70.447981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 208.964242] Lustre: Mounted lustre-client [ 214.417942] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 231.499230] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing check_logdir /tmp/testlogs/ [ 234.471641] Lustre: lustre-OST0000-osc-ffff902684a19800: disconnect after 23s idle [ 237.443638] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing yml_node [ 241.802295] Lustre: DEBUG MARKER: Client: 2.17.54.107 [ 243.799477] Lustre: DEBUG MARKER: MDS: 2.17.54.107 [ 246.630086] Lustre: DEBUG MARKER: OSS: 2.17.54.107 [ 248.085753] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 10 17:52:54 EDT 2026 [ 265.853912] Lustre: Unmounted lustre-client [ 296.684805] Key type lgssc unregistered [ 296.960776] LNet: 8393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 296.973899] LNetError: 8393:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 296.999468] LNet: Removed LNI 192.168.202.28@tcp [ 297.845339] Key type .llcrypt unregistered [ 297.848231] Key type ._llcrypt unregistered [ 324.186364] Key type ._llcrypt registered [ 324.194234] Key type .llcrypt registered [ 324.216683] LNet: 8973:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 324.216683] mask usage: [+|-] ... [ 324.352715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 324.367901] alg: No test for adler32 (adler32-zlib) [ 325.259678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 325.303807] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 325.319538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 329.361270] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 329.483278] LNet: 9270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.490718] LNetError: 9270:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 329.504240] LNet: Removed LNI 192.168.202.28@tcp [ 330.187302] Key type .llcrypt unregistered [ 330.190934] Key type ._llcrypt unregistered [ 335.868692] Key type ._llcrypt registered [ 335.870598] Key type .llcrypt registered [ 337.259375] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 17:54:23 (1786398863) [ 337.430464] LNet: 9645:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 337.430464] mask usage: [+|-] ... [ 337.684098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 337.704369] alg: No test for adler32 (adler32-zlib) [ 338.709648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.786771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.806239] LNet: 9688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.823616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.874467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 338.897270] LNet: 9695:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 345.706183] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 17:54:31 (1786398871) [ 346.087569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.109978] LNet: 10274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.138950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.238527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 351.843965] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 17:54:38 (1786398878) [ 352.148172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.177263] LNet: 10860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.195425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.282237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 357.850105] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 17:54:43 (1786398883) [ 358.044450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.063898] LNet: 11447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.078623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.136686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 363.992592] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 17:54:50 (1786398890) [ 364.331146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.359169] LNet: 12031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.397660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.467860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 364.875826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 371.389657] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 17:54:57 (1786398897) [ 371.643201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.679168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.778249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 372.315812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 372.934292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 380.059682] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 17:55:06 (1786398906) [ 380.123095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.151305] LNet: 13212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.156281] LNet: 13212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 380.186590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.293767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 386.745677] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 17:55:12 (1786398912) [ 387.152483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.231222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.345645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 387.405661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 388.168874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 388.223970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 388.799746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 388.846474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 389.572744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 389.666958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 390.286898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 390.325674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 390.878722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 390.956133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 391.674725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 391.742264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 392.379697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 392.436440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 392.953450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 393.023557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 393.778857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 393.864092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 394.580371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 394.634650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 395.298455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 395.362458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 396.208583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 396.308239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 404.292599] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 17:55:30 (1786398930) [ 404.615589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.671977] LNet: 14487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.683574] LNet: 14487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 404.721372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.813561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 404.860996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 411.070516] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 17:55:37 (1786398937) [ 411.340521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.374584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.470606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 411.514709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 417.932344] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 17:55:44 (1786398944) [ 418.118240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.169341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.229426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 418.265700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 424.628798] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 17:55:50 (1786398950) [ 424.949917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.012508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.103413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 425.162100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 431.341697] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 17:55:57 (1786398957) [ 431.499728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.571285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.650271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 431.689659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 439.120520] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 17:56:04 (1786398964) [ 439.319686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.356797] LNet: 17417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 439.365248] LNet: 17417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 439.408086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.547830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 439.611390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 448.381531] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 17:56:14 (1786398974) [ 448.683895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.728309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.808119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 449.258546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 456.643606] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 17:56:22 (1786398982) [ 456.976350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.019297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.159577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 457.351571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 464.016940] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 17:56:29 (1786398989) [ 464.114868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.153695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.332427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 464.564976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 471.506354] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 17:56:37 (1786398997) [ 471.568404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.613450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.028323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 472.281343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 478.501222] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 17:56:44 (1786399004) [ 478.743900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.773718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.823629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 478.854216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 485.035267] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 17:56:51 (1786399011) [ 485.153287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.177245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.252224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 485.295699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 490.825936] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 17:56:57 (1786399017) [ 491.273730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.326517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.386183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 497.390712] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 17:57:03 (1786399023) [ 497.675632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.730569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.809786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 504.182831] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 17:57:10 (1786399030) [ 504.524102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.544580] LNet: 22855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 504.548653] LNet: 22855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 504.569151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.663732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 505.078712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 512.098108] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 17:57:18 (1786399038) [ 512.233898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.297044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.354806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 512.379549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 518.922419] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 17:57:24 (1786399044) [ 519.143723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.191264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.256189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 519.289775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 526.166882] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 17:57:32 (1786399052) [ 526.385122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.403319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.433298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 526.456739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 534.216482] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 17:57:39 (1786399059) [ 534.731930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.824839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.962793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 535.024290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 544.214657] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 17:57:49 (1786399069) [ 544.485699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 544.528554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.613751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 544.707392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 544.728178] LNetError: 25802:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 544.776237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 544.808419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 552.395894] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 17:57:58 (1786399078) [ 552.589188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.654835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.771415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.0VCEtNjpy1 [ 552.890507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.0VCEtNjpy1 [ 552.988020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.0VCEtNjpy1 [ 559.534514] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 17:58:05 (1786399085) [ 559.825451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.946113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.023884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 560.061411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 560.098244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 566.779731] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 17:58:12 (1786399092) [ 566.858314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.890617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.947147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 568.420466] LNet: Message forwarding will be enabled in 2s [ 571.489919] LNet: Message forwarding enabled [ 572.054951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 572.077424] LNet: Message forwarding will stop in 2s to 6s [ 576.364555] LNet: Message forwarding disabled [ 576.661643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 578.456705] LNet: Message forwarding will be enabled in 2s [ 582.502477] LNet: Message forwarding enabled [ 585.558877] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 17:58:31 (1786399111) [ 585.662945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.481619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.607692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 594.002655] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 17:58:39 (1786399119) [ 594.107949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.150350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.203765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 594.243958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 594.279412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 594.336781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 594.382619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 594.417146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 594.463827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 594.512867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 594.552539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 594.615111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 594.660501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 594.697483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 594.734890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 594.770495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 594.812295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 594.884261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 594.967918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 595.030515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 595.113866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 595.147636] LNetError: 28872:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 601.931765] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 17:58:47 (1786399127) [ 602.318486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.372550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.488993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 609.733477] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 17:58:55 (1786399135) [ 611.729330] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg228-server [ 613.547156] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 17:58:59 (1786399139) [ 613.870777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.895878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.935358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 613.960980] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 615.425163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[29-33]@tcp [ 615.446262] LNet: 30589:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 615.497030] LNetError: 30379:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.30@tcp is being used as a gateway but routing feature is not turned on [ 615.517233] LNetError: 30379:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.30@tcp has gone from up to down [ 616.481766] LNetError: 30379:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.31@tcp is being used as a gateway but routing feature is not turned on [ 616.502118] LNetError: 30379:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 617.508377] LNetError: 30379:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.33@tcp is being used as a gateway but routing feature is not turned on [ 617.522361] LNetError: 30379:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 619.558929] LNetError: 30379:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.29@tcp is being used as a gateway but routing feature is not turned on [ 619.571691] LNetError: 30379:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 622.467452] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 17:59:08 (1786399148) [ 623.651740] LNetError: 30379:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.29@tcp is being used as a gateway but routing feature is not turned on [ 623.675882] LNetError: 30379:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 625.098515] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg228-server [ 627.914433] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 17:59:12 (1786399152) [ 628.031725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 628.069457] LNetError: 31520:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 628.112736] LNet: Removed LNI 192.168.202.28@tcp [ 628.166752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.295262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 628.316237] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 629.124791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[29-34/2]@tcp [ 629.157869] LNet: 31643:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 629.171177] LNet: 31643:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 629.214396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[29-34/2]@tcp [ 636.239750] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 17:59:22 (1786399162) [ 636.463473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.541655] LNet: 32232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.556996] LNet: 32232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 636.573619] LNetError: 32232:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.609365] LNet: Removed LNI 192.168.202.28@tcp [ 637.688208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.776699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 637.908087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.954821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.031470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.120299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.151836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.189706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 638.253911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.284115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.334852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.407880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.440254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.513692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 638.628786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.672636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.748816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.836144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.882505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.932502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 639.043296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.091098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.172476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 639.230289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.262521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.297731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 639.377419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.402420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.462440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 639.567545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.611620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.661603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 639.714805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.749079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.810327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 646.874387] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 17:59:32 (1786399172) [ 648.651231] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg228-server [ 650.693661] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 17:59:36 (1786399176) [ 652.844540] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg228-server [ 654.370149] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 17:59:40 (1786399180) [ 655.505469] Key type .llcrypt unregistered [ 655.509222] Key type ._llcrypt unregistered [ 662.927104] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 663.038624] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 663.048822] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 663.065078] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 665.731150] Key type ._llcrypt registered [ 665.734206] Key type .llcrypt registered [ 665.800297] LNet: 33917:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 665.800297] mask usage: [+|-] ... [ 666.039282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 666.057300] alg: No test for adler32 (adler32-zlib) [ 667.082248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.162293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 667.183339] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 667.782334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 667.819765] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 670.428212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 670.449413] LNet: Removed LNI 10.1.2.3@tcp [ 670.604658] LNet: 34359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 670.615617] LNetError: 34359:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 670.633102] LNet: Removed LNI 192.168.202.28@tcp [ 671.396398] Key type .llcrypt unregistered [ 671.398333] Key type ._llcrypt unregistered [ 684.220233] Key type ._llcrypt registered [ 684.222577] Key type .llcrypt registered [ 685.614641] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 18:00:11 (1786399211) [ 686.571496] Key type .llcrypt unregistered [ 686.576657] Key type ._llcrypt unregistered [ 692.665309] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 692.930488] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 692.936080] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 692.942664] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 695.435975] Key type ._llcrypt registered [ 695.437670] Key type .llcrypt registered [ 695.462304] LNet: 35393:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 695.462304] mask usage: [+|-] ... [ 695.618920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 695.633317] alg: No test for adler32 (adler32-zlib) [ 696.538236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.597252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 696.623503] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 697.276681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 697.292444] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 699.502561] LNet: 35853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.506529] LNetError: 35853:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 699.509288] LNet: Removed LNI 10.1.2.3@tcp [ 699.518320] LNet: Removed LNI 192.168.202.28@tcp [ 700.084537] Key type .llcrypt unregistered [ 700.085848] Key type ._llcrypt unregistered [ 711.279776] Key type ._llcrypt registered [ 711.281855] Key type .llcrypt registered [ 712.441713] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 18:00:38 (1786399238) [ 712.893362] Key type .llcrypt unregistered [ 712.895235] Key type ._llcrypt unregistered [ 718.325982] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 718.425366] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 718.439373] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 718.447399] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 721.024755] Key type ._llcrypt registered [ 721.035382] Key type .llcrypt registered [ 721.104415] LNet: 36901:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 721.104415] mask usage: [+|-] ... [ 721.314766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 721.339022] alg: No test for adler32 (adler32-zlib) [ 722.246888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.305115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 722.326207] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 723.958814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 723.978630] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 725.926228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 725.945111] LNet: Removed LNI 10.1.2.3@tcp [ 725.958276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 725.988548] LNet: Removed LNI 10.1.2.31@tcp [ 726.000528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 726.024495] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 726.651745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 726.666748] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 727.427299] LNet: 37784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.430922] LNetError: 37784:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 727.435555] LNet: Removed LNI 10.1.2.31@tcp [ 727.445909] LNet: Removed LNI 10.1.2.3@tcp [ 727.853388] Key type .llcrypt unregistered [ 727.855449] Key type ._llcrypt unregistered [ 740.845278] Key type ._llcrypt registered [ 740.849723] Key type .llcrypt registered [ 742.104616] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 18:01:08 (1786399268) [ 742.426234] LNet: 38590:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 742.426234] mask usage: [+|-] ... [ 742.612210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 742.628491] alg: No test for adler32 (adler32-zlib) [ 743.649957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.714483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 743.745379] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 744.572880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 744.610537] LNet: 38750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.621597] LNetError: 38750:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 744.634647] LNet: Removed LNI 192.168.202.28@tcp [ 744.649505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.691492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.28@tcp [ 744.712505] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 745.462472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.28@tcp [ 745.489206] LNet: Removed LNI 192.168.202.28@tcp [ 745.551402] LNet: 38885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.169787] Key type .llcrypt unregistered [ 746.174931] Key type ._llcrypt unregistered [ 759.188207] Key type ._llcrypt registered [ 759.190196] Key type .llcrypt registered [ 760.352187] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 18:01:26 (1786399286) [ 762.200726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg228-server [ 764.042256] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 18:01:29 (1786399289) [ 764.861343] Key type .llcrypt unregistered [ 764.863261] Key type ._llcrypt unregistered [ 770.577244] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 770.808202] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 770.814304] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 770.825057] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 773.446451] Key type ._llcrypt registered [ 773.450745] Key type .llcrypt registered [ 773.493679] LNet: 40263:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 773.493679] mask usage: [+|-] ... [ 773.651741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 773.666736] alg: No test for adler32 (adler32-zlib) [ 774.574667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.640924] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 775.502802] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 777.545049] LNet: 40723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.555953] LNetError: 40723:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.558044] LNet: Removed LNI 192.168.202.28@tcp1 [ 777.578153] LNet: Removed LNI 10.1.2.3@tcp2 [ 778.159456] Key type .llcrypt unregistered [ 778.162211] Key type ._llcrypt unregistered [ 789.517755] Key type ._llcrypt registered [ 789.519960] Key type .llcrypt registered [ 790.735962] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 18:01:56 (1786399316) [ 791.017800] LNet: 41527:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 791.017800] mask usage: [+|-] ... [ 791.104314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 791.115789] alg: No test for adler32 (adler32-zlib) [ 792.050543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.086810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 792.103120] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 792.673234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.688300] LNet: 41692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.692296] LNetError: 41692:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.704745] LNet: Removed LNI 192.168.202.28@tcp [ 792.719814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.786996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 792.827227] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 799.316327] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 18:02:05 (1786399325) [ 799.478440] LNet: 42284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.489716] LNetError: 42284:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.526205] LNet: Removed LNI 192.168.202.28@tcp [ 800.216384] Key type .llcrypt unregistered [ 800.218292] Key type ._llcrypt unregistered [ 807.104923] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 807.323903] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 807.333377] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 807.335873] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 810.045225] Key type ._llcrypt registered [ 810.046947] Key type .llcrypt registered [ 810.105681] LNet: 42561:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 810.105681] mask usage: [+|-] ... [ 810.256413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.270527] alg: No test for adler32 (adler32-zlib) [ 811.256699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.323434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.348765] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 812.276324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 812.295863] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 814.440493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.457200] LNet: 43027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.460405] LNetError: 43027:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 814.463082] LNet: Removed LNI 10.1.2.3@tcp [ 814.477654] LNet: Removed LNI 192.168.202.28@tcp [ 814.502946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.550574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 814.570708] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 814.574731] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 815.264429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.290395] LNet: 43054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.300746] LNetError: 43054:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 815.306967] LNetError: 43054:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 815.311884] LNet: Removed LNI 10.1.2.3@tcp [ 815.325645] LNet: Removed LNI 192.168.202.28@tcp [ 815.348554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.419775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 815.450706] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 815.455127] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 815.996796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 816.029816] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 817.206148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.226151] LNet: 43216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.234823] LNetError: 43216:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 817.246553] LNetError: 43216:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 817.265366] LNet: Removed LNI 10.1.2.3@tcp [ 817.272514] LNet: Removed LNI 192.168.202.28@tcp [ 817.285501] LNet: Removed LNI 192.168.202.28@tcp2 [ 817.316329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.390575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 817.430245] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 817.435683] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 817.442863] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 824.419667] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 18:02:30 (1786399350) [ 824.618181] LNet: 43810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.628327] LNetError: 43810:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 824.635692] LNetError: 43810:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 824.645612] LNet: Removed LNI 10.1.2.3@tcp [ 824.657712] LNet: Removed LNI 192.168.202.28@tcp [ 824.673801] LNet: Removed LNI 192.168.202.28@tcp2 [ 825.309943] Key type .llcrypt unregistered [ 825.320438] Key type ._llcrypt unregistered [ 832.613922] Key type ._llcrypt registered [ 832.617203] Key type .llcrypt registered [ 832.656926] LNet: 44039:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 832.656926] mask usage: [+|-] ... [ 832.834519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 832.849984] alg: No test for adler32 (adler32-zlib) [ 833.813432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.860644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 833.880339] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 834.793626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 834.824586] LNet: 44208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.838833] LNetError: 44208:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.863968] LNet: Removed LNI 192.168.202.28@tcp [ 834.890801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.949919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 834.982953] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 835.514322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.550657] LNet: 44239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.558268] LNetError: 44239:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.573728] LNet: Removed LNI 192.168.202.28@tcp [ 835.604715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.663949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 835.690776] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 836.265792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 836.300783] LNet: Removed LNI 192.168.202.28@tcp [ 836.319869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.380075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 836.408333] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 836.860810] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 837.013405] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 837.021293] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 837.041834] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 839.507616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 839.548175] LNet: 44347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.552621] LNet: 44347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 839.560450] LNetError: 44347:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.569030] LNetError: 44347:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 839.579358] LNet: Removed LNI 192.168.202.28@tcp [ 839.601524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.676809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 839.700248] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 845.923562] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 18:02:52 (1786399372) [ 846.291401] LNet: 44945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.300380] LNetError: 44945:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.321253] LNet: Removed LNI 192.168.202.28@tcp [ 846.881563] Key type .llcrypt unregistered [ 846.883283] Key type ._llcrypt unregistered [ 853.142742] Key type ._llcrypt registered [ 853.146270] Key type .llcrypt registered [ 853.196909] LNet: 45170:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 853.196909] mask usage: [+|-] ... [ 853.366522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 853.385963] alg: No test for adler32 (adler32-zlib) [ 854.312975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.424343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 854.473121] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 855.515769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 855.547398] LNet: 45343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.561417] LNetError: 45343:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.574823] LNet: Removed LNI 192.168.202.28@tcp [ 855.607387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.673746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 855.711887] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 862.053545] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 18:03:08 (1786399388) [ 862.194610] LNet: 45926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.220252] LNetError: 45926:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.258935] LNet: Removed LNI 192.168.202.28@tcp [ 862.900349] Key type .llcrypt unregistered [ 862.904444] Key type ._llcrypt unregistered [ 868.705848] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 868.935132] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 868.941517] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 868.946180] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 871.632731] Key type ._llcrypt registered [ 871.634383] Key type .llcrypt registered [ 871.678803] LNet: 46211:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 871.678803] mask usage: [+|-] ... [ 871.905238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 871.921291] alg: No test for adler32 (adler32-zlib) [ 872.803205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.845961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 872.858956] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 873.746884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.784892] LNet: 46384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.791409] LNetError: 46384:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.822606] LNet: Removed LNI 192.168.202.28@tcp [ 873.855869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.961631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 874.014721] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 882.584372] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 18:03:28 (1786399408) [ 882.697477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.738433] LNet: 46977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.752588] LNetError: 46977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.784872] LNet: Removed LNI 192.168.202.28@tcp [ 882.831774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.946849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 889.499547] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 18:03:35 (1786399415) [ 889.614769] LNet: 47557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.485638] Key type .llcrypt unregistered [ 890.490401] Key type ._llcrypt unregistered [ 897.435641] Key type ._llcrypt registered [ 897.437845] Key type .llcrypt registered [ 897.487362] LNet: 47791:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 897.487362] mask usage: [+|-] ... [ 897.641790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 897.670863] alg: No test for adler32 (adler32-zlib) [ 898.593634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.634608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 898.656119] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 899.577197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.604357] LNet: 47963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.607935] LNetError: 47963:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.622614] LNet: Removed LNI 192.168.202.28@tcp [ 899.642517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.718751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 899.767723] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 907.264610] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 18:03:52 (1786399432) [ 907.418001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.457345] LNet: 48553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.475382] LNetError: 48553:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.508191] LNet: Removed LNI 192.168.202.28@tcp [ 907.571467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.708994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 914.566324] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 18:04:00 (1786399440) [ 916.111284] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 917.572663] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 18:04:03 (1786399443) [ 919.163059] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 920.699638] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 18:04:06 (1786399446) [ 922.392858] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 923.961745] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 18:04:10 (1786399450) [ 925.120393] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 926.465702] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 18:04:12 (1786399452) [ 927.866394] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 930.259580] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 18:04:15 (1786399455) [ 930.454531] LNet: 50608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.137443] Key type .llcrypt unregistered [ 931.141847] Key type ._llcrypt unregistered [ 937.024613] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 937.189767] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 937.192291] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 937.195529] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 939.739366] Key type ._llcrypt registered [ 939.741309] Key type .llcrypt registered [ 939.779463] LNet: 50895:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 939.779463] mask usage: [+|-] ... [ 939.896173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 939.906366] alg: No test for adler32 (adler32-zlib) [ 940.759788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.820793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 940.838461] LNet: Added LNI 192.168.202.28@tcp [8/512/0/180] [ 940.842932] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 940.845676] LNet: 50933:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 940.937547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 940.952364] LNet: 50952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.958315] LNetError: 50952:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 940.962704] LNet: Removed LNI 10.1.2.3@tcp [ 940.965204] LNetError: 50952:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.981756] LNet: Removed LNI 192.168.202.28@tcp [ 941.004083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.088550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 941.141615] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 941.223979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.253274] LNet: 50976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.257834] LNetError: 50976:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.282868] LNet: Removed LNI 192.168.202.28@tcp [ 941.310498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.398435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 941.439420] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 941.443338] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 941.445144] LNet: 50984:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 941.562764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.594189] LNet: 51002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.598521] LNetError: 51002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 941.602619] LNet: Removed LNI 10.1.2.3@tcp [ 941.604321] LNetError: 51002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.613919] LNet: Removed LNI 192.168.202.28@tcp [ 941.627821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.706759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 941.738417] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 941.744199] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 941.751215] LNet: 51010:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 941.936957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.948720] LNet: 51036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.951558] LNetError: 51036:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 941.953958] LNet: Removed LNI 10.1.2.3@tcp [ 941.955164] LNetError: 51036:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.963471] LNet: Removed LNI 192.168.202.28@tcp [ 941.978175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.029173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 942.070217] LNet: Added LNI 192.168.202.28@tcp [8/256/0/182] [ 942.074065] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 942.178932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 942.199817] LNet: 51064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.211238] LNetError: 51064:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 942.225205] LNet: Removed LNI 10.1.2.3@tcp [ 942.227768] LNetError: 51064:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.250776] LNet: Removed LNI 192.168.202.28@tcp [ 942.271937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.336523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 942.360281] LNet: Added LNI 192.168.202.28@tcp [8/256/0/182] [ 942.369424] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 942.376142] LNet: 51072:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 942.510304] LNet: Removed LNI 10.1.2.3@tcp [ 942.528840] LNet: Removed LNI 192.168.202.28@tcp [ 943.032311] Key type .llcrypt unregistered [ 943.035787] Key type ._llcrypt unregistered [ 954.378816] Key type ._llcrypt registered [ 954.383984] Key type .llcrypt registered [ 956.470575] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 18:04:41 (1786399481) [ 956.688595] LNet: 51887:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 956.688595] mask usage: [+|-] ... [ 956.784333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 956.794341] alg: No test for adler32 (adler32-zlib) [ 957.732896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.853848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 957.916424] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 959.218429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.28 [ 959.234748] LNet: 52049:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 959.241572] LNet: Removed LNI 192.168.202.28@tcp [ 966.708319] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 18:04:52 (1786399492) [ 967.173320] LNet: 52619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.221645] LNetError: 52619:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.296657] LNet: Removed LNI 192.168.202.28@tcp [ 968.512697] Key type .llcrypt unregistered [ 968.517069] Key type ._llcrypt unregistered [ 976.521748] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 976.918225] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 978.756445] Key type ._llcrypt registered [ 978.757612] Key type .llcrypt registered [ 978.788991] LNet: 52894:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 978.788991] mask usage: [+|-] ... [ 978.987876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 979.016148] alg: No test for adler32 (adler32-zlib) [ 980.398617] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 988.554741] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 18:05:14 (1786399514) [ 988.838157] LNet: 53527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.843524] LNetError: 53527:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 988.868266] LNet: Removed LNI 10.1.2.3@tcp [ 989.474736] Key type .llcrypt unregistered [ 989.476810] Key type ._llcrypt unregistered [ 996.218166] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 996.480452] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 996.487624] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 996.496706] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 999.485770] Key type ._llcrypt registered [ 999.487992] Key type .llcrypt registered [ 999.538870] LNet: 53800:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 999.538870] mask usage: [+|-] ... [ 999.734929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 999.755256] alg: No test for adler32 (adler32-zlib) [ 1000.756599] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1007.422227] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 18:05:33 (1786399533) [ 1009.245235] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1011.439760] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 18:05:37 (1786399537) [ 1011.671129] LNet: 54711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.677108] LNetError: 54711:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1011.687146] LNet: Removed LNI 10.1.2.3@tcp [ 1012.723650] Key type .llcrypt unregistered [ 1012.725803] Key type ._llcrypt unregistered [ 1021.016228] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1021.264774] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1021.271259] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1021.277899] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1024.186057] Key type ._llcrypt registered [ 1024.187978] Key type .llcrypt registered [ 1024.220435] LNet: 54986:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1024.220435] mask usage: [+|-] ... [ 1024.404499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1024.420814] alg: No test for adler32 (adler32-zlib) [ 1025.412235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.499325] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1035.811386] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 18:06:01 (1786399561) [ 1036.461342] LNet: 55943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.473951] LNetError: 55943:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1036.496344] LNet: Removed LNI 10.1.2.3@tcp [ 1037.376399] Key type .llcrypt unregistered [ 1037.381672] Key type ._llcrypt unregistered [ 1044.071639] Key type ._llcrypt registered [ 1044.073578] Key type .llcrypt registered [ 1044.111392] LNet: 56167:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1044.111392] mask usage: [+|-] ... [ 1044.342946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1044.369393] alg: No test for adler32 (adler32-zlib) [ 1045.295457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1045.318831] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1052.708979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1053.571358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_interrupt -r 1 [ 1053.614440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_interrupt -r 1 [ 1053.668037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1053.732326] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1054.126537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1054.155525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1054.178653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1054.204757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1054.234236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1054.282176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1054.298616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1054.313228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1054.322349] LNet: 56382:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1054.330942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1054.348739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1054.369155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1054.391332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1054.417191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1054.465463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1054.488151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1054.854568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_dropped -r 1 [ 1054.882474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_dropped -r 1 [ 1054.906543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1054.929981] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1055.132545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1055.161869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1055.190537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1055.216365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1055.246680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1055.266563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1055.296790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1055.340511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1055.353416] LNet: 56491:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1055.370896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1055.393706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1055.423956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1055.445756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1055.467852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1055.488501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1055.507254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1055.804825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_aborted -r 1 [ 1055.823976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_aborted -r 1 [ 1055.849800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1055.998697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1056.025815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1056.040305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1056.052888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1056.067683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1056.082405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1056.104521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1056.126557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1056.146065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1056.166587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1056.185675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1056.211824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1056.244838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1056.266633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1056.310793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1056.938204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_no_route -r 1 [ 1056.983792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_no_route -r 1 [ 1057.007519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1057.017712] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1057.019992] LNet: Skipped 1 previous similar message [ 1057.136913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1057.159279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1057.189828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1057.212617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1057.249296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1057.275812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1057.304390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1057.331472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1057.350302] LNet: 56709:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1057.353821] LNet: 56709:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1057.367660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1057.399386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1057.450966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1057.482737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1057.539293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1057.583650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1057.651204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1058.137518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_timeout -r 1 [ 1058.163284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_timeout -r 1 [ 1058.213335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1058.452837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1058.486427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1058.514641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1058.553260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1058.603835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1058.637829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1058.663867] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1058.679635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1058.716280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1058.736454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1058.761773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1058.821791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1058.874512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1058.909122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1058.937454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1058.968175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1059.400199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_error -r 1 [ 1059.432848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_error -r 1 [ 1059.467247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1059.489802] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1059.494703] LNet: Skipped 1 previous similar message [ 1059.725538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1059.758840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1059.789790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1059.820286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1059.881308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1059.950415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1059.982534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1060.011225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1060.025588] LNet: 56928:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1060.028855] LNet: 56928:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1060.051235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1060.081799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1060.099148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1060.125513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1060.152826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1060.184530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1060.210646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1060.386443] LNet: 56969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.392476] LNetError: 56969:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.415217] LNet: Removed LNI 192.168.202.28@tcp [ 1061.796423] Key type .llcrypt unregistered [ 1061.798121] Key type ._llcrypt unregistered [ 1078.571221] Key type ._llcrypt registered [ 1078.574815] Key type .llcrypt registered [ 1079.700667] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 18:06:45 (1786399605) [ 1080.539496] Key type .llcrypt unregistered [ 1080.541381] Key type ._llcrypt unregistered [ 1086.269858] Key type ._llcrypt registered [ 1086.271559] Key type .llcrypt registered [ 1086.305840] LNet: 58008:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1086.305840] mask usage: [+|-] ... [ 1086.452519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1086.468578] alg: No test for adler32 (adler32-zlib) [ 1087.340696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1087.360926] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1095.881536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1105.698909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1105.760384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1105.803017] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1107.165859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_interrupt -r 1 [ 1107.194868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_interrupt -r 1 [ 1107.230266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_interrupt -r 1 [ 1107.258966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_interrupt -r 1 [ 1107.287361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_interrupt -r 1 [ 1107.319782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_interrupt -r 1 [ 1107.354515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_interrupt -r 1 [ 1107.393669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_interrupt -r 1 [ 1107.443490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1107.488302] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1107.513763] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1107.768856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.800586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1107.822589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.854708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.886322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.920657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.974041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.038877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.073378] LNet: 58438:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.107500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.168475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.209977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.244208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.269616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.298476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.324931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1108.774312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_dropped -r 1 [ 1108.802711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_dropped -r 1 [ 1108.833332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_dropped -r 1 [ 1108.866455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_dropped -r 1 [ 1108.929481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_dropped -r 1 [ 1108.971871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_dropped -r 1 [ 1108.998121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_dropped -r 1 [ 1109.031314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_dropped -r 1 [ 1109.083904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1109.103614] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1109.112502] LNet: Skipped 2 previous similar messages [ 1109.318824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.338677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1109.352542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.371352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.386988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.401703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.417486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.437789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.450411] LNet: 58559:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1109.471202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.498591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.533793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.557878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.580432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.612860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.640069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1110.192243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_aborted -r 1 [ 1110.250206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_aborted -r 1 [ 1110.281839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_aborted -r 1 [ 1110.320313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_aborted -r 1 [ 1110.357983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_aborted -r 1 [ 1110.392879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_aborted -r 1 [ 1110.431624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_aborted -r 1 [ 1110.455736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_aborted -r 1 [ 1110.491308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1110.520064] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1110.522887] LNet: Skipped 2 previous similar messages [ 1110.761466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.789454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1110.815902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.831305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.863083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.911061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.974269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.997443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1111.023512] LNet: 58681:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1111.049823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1111.094874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1111.160276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1111.179662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1111.210705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1111.236247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.261871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1111.692503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_no_route -r 1 [ 1111.741804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_no_route -r 1 [ 1111.780868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_no_route -r 1 [ 1111.814662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_no_route -r 1 [ 1111.853528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_no_route -r 1 [ 1111.883602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_no_route -r 1 [ 1111.906652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_no_route -r 1 [ 1111.946062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_no_route -r 1 [ 1112.019817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1112.261385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1112.294601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1112.324568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1112.351442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1112.382835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1112.407854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1112.440837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1112.477568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1112.507430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1112.541819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1112.622465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.675253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.723135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.779946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.825877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1113.283154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_timeout -r 1 [ 1113.317787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_timeout -r 1 [ 1113.359961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_timeout -r 1 [ 1113.415931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_timeout -r 1 [ 1113.509574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_timeout -r 1 [ 1113.552806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_timeout -r 1 [ 1113.606852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_timeout -r 1 [ 1113.635877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_timeout -r 1 [ 1113.669632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1113.694211] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1113.699449] LNet: Skipped 5 previous similar messages [ 1113.898424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.936754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1113.962547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1113.993932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.026671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.066886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.082860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.100971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.121354] LNet: 58923:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1114.129857] LNet: 58923:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1114.177106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.220158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1114.266851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1114.320454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1114.382392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.431298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1114.492875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1114.931883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_error -r 1 [ 1114.966664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_error -r 1 [ 1115.003260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_error -r 1 [ 1115.024352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_error -r 1 [ 1115.054848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_error -r 1 [ 1115.091945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_error -r 1 [ 1115.118802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_error -r 1 [ 1115.147689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_error -r 1 [ 1115.177820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1115.451424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1115.477275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1115.503048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1115.529688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1115.544378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1115.559438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1115.604203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1115.659429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1115.708174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1115.747568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1115.777326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.809087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.839607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.857193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1115.877844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1121.901760] LNet: 59122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1121.910231] LNetError: 59122:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1121.919726] LNet: Removed LNI 192.168.202.28@tcp [ 1121.937841] LNet: Removed LNI 192.168.202.28@tcp1 [ 1122.485455] Key type .llcrypt unregistered [ 1122.487558] Key type ._llcrypt unregistered [ 1141.233553] Key type ._llcrypt registered [ 1141.236545] Key type .llcrypt registered [ 1142.607032] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 18:07:48 (1786399668) [ 1143.639442] Key type .llcrypt unregistered [ 1143.641348] Key type ._llcrypt unregistered [ 1149.206041] Key type ._llcrypt registered [ 1149.207942] Key type .llcrypt registered [ 1149.252981] LNet: 60158:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1149.252981] mask usage: [+|-] ... [ 1149.476375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1149.508044] alg: No test for adler32 (adler32-zlib) [ 1150.420748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1150.440196] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1158.213424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1159.191979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_dropped -r 1 [ 1159.235421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_dropped -r 1 [ 1159.297045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1159.314371] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1159.604865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1159.657860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1159.704728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1159.768518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1159.841353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1159.902662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1159.948740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1159.980599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1160.006833] LNet: 60372:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1160.041412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1160.084354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1160.132213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1160.209342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1160.253546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1160.300598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1160.354875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1160.988617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_error -r 1 [ 1161.021978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_error -r 1 [ 1161.067625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1161.089787] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1161.290100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.317986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1161.356758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.401954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.449655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.488502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.541315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1161.572190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1161.601366] LNet: 60481:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1161.622345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1161.653811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1161.688761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1161.726513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1161.774631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1161.817294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1161.862775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1162.391873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_timeout -r 1 [ 1162.452382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_timeout -r 1 [ 1162.514547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1162.556309] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1162.786590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1162.809808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1162.834356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1162.872447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1162.899990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1162.928468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1162.994368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.024199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.040553] LNet: 60591:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.063592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.090696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.141444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.195279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.234233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.283284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.317537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1163.642374] LNet: 60632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.650563] LNetError: 60632:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1163.667648] LNet: Removed LNI 192.168.202.28@tcp [ 1164.072449] Key type .llcrypt unregistered [ 1164.074980] Key type ._llcrypt unregistered [ 1181.872921] Key type ._llcrypt registered [ 1181.876621] Key type .llcrypt registered [ 1183.123362] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 18:08:29 (1786399709) [ 1183.910263] Key type .llcrypt unregistered [ 1183.911954] Key type ._llcrypt unregistered [ 1191.157231] Key type ._llcrypt registered [ 1191.161447] Key type .llcrypt registered [ 1191.235546] LNet: 61670:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1191.235546] mask usage: [+|-] ... [ 1191.422811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1191.437925] alg: No test for adler32 (adler32-zlib) [ 1192.334150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1192.400248] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1199.392893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1210.241272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1210.264566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1210.287698] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1211.722684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_dropped -r 1 [ 1211.812374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e remote_dropped -r 1 [ 1211.859578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_dropped -r 1 [ 1211.952833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e remote_dropped -r 1 [ 1212.005824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e remote_dropped -r 1 [ 1212.072733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e remote_dropped -r 1 [ 1212.132572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e remote_dropped -r 1 [ 1212.194819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e remote_dropped -r 1 [ 1212.235913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1212.258952] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1212.265131] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp [ 1212.550485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.575777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1212.620131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.662657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1212.690337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1212.722945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1212.757981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1212.799910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.832819] LNet: 62099:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1212.856957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1212.898785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.935643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.971313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1213.001610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.061988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1213.105843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1213.151749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.719848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_error -r 1 [ 1213.765723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e remote_error -r 1 [ 1213.814320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_error -r 1 [ 1213.852978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e remote_error -r 1 [ 1213.895575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e remote_error -r 1 [ 1213.942856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e remote_error -r 1 [ 1214.002198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e remote_error -r 1 [ 1214.034730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e remote_error -r 1 [ 1214.106431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1214.128301] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1214.137642] LNet: Skipped 2 previous similar messages [ 1214.474692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1214.523653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1214.569668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1214.599971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1214.634318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1214.700890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1214.760866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1214.813966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1214.862574] LNet: 62223:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1214.910795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1214.959372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1214.998695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1215.030648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1215.062560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.095901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1215.135187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1215.190861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.706564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_timeout -r 1 [ 1215.733849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e remote_timeout -r 1 [ 1215.761813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_timeout -r 1 [ 1215.788187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e remote_timeout -r 1 [ 1215.814932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e remote_timeout -r 1 [ 1215.834389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e remote_timeout -r 1 [ 1215.854175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e remote_timeout -r 1 [ 1215.886766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e remote_timeout -r 1 [ 1215.938479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.128@tcp [ 1215.954795] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1216.170155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1216.222522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1216.274443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1216.306334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1216.349098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1216.393653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1216.420052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1216.456644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1216.474354] LNet: 62346:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1216.514287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1216.569694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1216.611322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1216.663313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1216.737048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1216.828299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1216.887618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1216.915952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.595821] LNet: 62424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.608440] LNetError: 62424:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.618172] LNet: Removed LNI 192.168.202.28@tcp [ 1221.640430] LNet: Removed LNI 192.168.202.28@tcp1 [ 1222.071501] Key type .llcrypt unregistered [ 1222.075848] Key type ._llcrypt unregistered [ 1238.709326] Key type ._llcrypt registered [ 1238.711390] Key type .llcrypt registered [ 1239.876523] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 18:09:25 (1786399765) [ 1241.272828] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1243.086587] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 18:09:29 (1786399769) [ 1243.884300] Key type .llcrypt unregistered [ 1243.887644] Key type ._llcrypt unregistered [ 1250.829488] Key type ._llcrypt registered [ 1250.831524] Key type .llcrypt registered [ 1250.866864] LNet: 63752:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1250.866864] mask usage: [+|-] ... [ 1251.076738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1251.104690] alg: No test for adler32 (adler32-zlib) [ 1252.040027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1252.079348] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1259.335860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1260.134273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e network_timeout -r 1 [ 1260.173525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e network_timeout -r 1 [ 1260.252215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1260.285388] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1260.513879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1260.558376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1260.598903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1260.646796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1260.691078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1260.712709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1260.738565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1260.783691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1260.808193] LNet: 63966:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1260.840687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1260.896565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1260.941488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1260.988680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1261.034800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1261.075313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1261.338520] LNet: 64005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1261.342882] LNetError: 64005:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1261.356901] LNet: Removed LNI 192.168.202.28@tcp [ 1261.893714] Key type .llcrypt unregistered [ 1261.905650] Key type ._llcrypt unregistered [ 1282.179655] Key type ._llcrypt registered [ 1282.181543] Key type .llcrypt registered [ 1282.234536] LNet: 64455:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1282.234536] mask usage: [+|-] ... [ 1282.377283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1282.390409] alg: No test for adler32 (adler32-zlib) [ 1283.392792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1283.417139] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1292.380565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1302.350867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1302.419917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1302.459956] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1303.904345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e network_timeout -r 1 [ 1303.955696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e network_timeout -r 1 [ 1304.002511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e network_timeout -r 1 [ 1304.052273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e network_timeout -r 1 [ 1304.087374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e network_timeout -r 1 [ 1304.122459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e network_timeout -r 1 [ 1304.164820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e network_timeout -r 1 [ 1304.214163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e network_timeout -r 1 [ 1304.288461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1304.322334] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1304.554226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1304.578473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1304.603557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1304.648396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1304.688227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1304.736077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1304.794283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1304.857785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1304.875480] LNet: 64885:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1304.904713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1304.929123] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1304.932409] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp [ 1304.934227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1304.965803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.006279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1305.053282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.100889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1309.509698] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1309.825801] LNet: 64958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.840945] LNetError: 64958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1309.848717] LNet: Removed LNI 192.168.202.28@tcp [ 1309.873433] LNet: Removed LNI 192.168.202.28@tcp1 [ 1310.351347] Key type .llcrypt unregistered [ 1310.357068] Key type ._llcrypt unregistered [ 1330.457233] Key type ._llcrypt registered [ 1330.462821] Key type .llcrypt registered [ 1332.219566] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 18:10:57 (1786399857) [ 1332.428842] LNet: 65757:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1332.428842] mask usage: [+|-] ... [ 1332.716534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1332.753821] alg: No test for adler32 (adler32-zlib) [ 1333.961265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1334.038648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1334.075895] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1335.241537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1335.283047] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1336.636085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1336.741585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1336.790502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1336.829458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1336.862718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1336.896216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1336.928585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.28@tcp [ 1336.947787] LNetError: 66059:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.28@tcp to recovery [ 1337.056000] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1337.058663] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1339.104923] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1339.549804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1343.199583] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1344.369847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1344.397338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1344.440141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1344.456627] LNet: 66166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1344.461780] LNetError: 66166:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.503219] LNet: Removed LNI 192.168.202.28@tcp [ 1345.505620] LNetError: 66166:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.516587] LNet: Removed LNI 192.168.202.28@tcp1 [ 1345.539829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1345.605932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1345.663120] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1346.769520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1346.801075] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1347.718077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1347.749417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1347.809765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1347.844521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1347.865431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1347.892677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1347.925500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1347.970891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.28@tcp [ 1347.994783] LNetError: 66438:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.28@tcp to recovery [ 1348.642602] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1351.591452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1354.783430] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1354.791684] LNet: Skipped 1 previous similar message [ 1359.579203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1359.615347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1359.645750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1366.361083] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 18:11:32 (1786399892) [ 1366.737611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1366.770893] LNet: 67156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1366.776428] LNetError: 67156:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1366.784597] LNet: Removed LNI 192.168.202.28@tcp [ 1366.798105] LNet: Removed LNI 192.168.202.28@tcp1 [ 1366.823573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1366.875390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1366.908950] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1367.615479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1367.635613] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1368.342956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1368.453326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1368.491373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1368.542898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1368.580975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1368.609410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1368.636431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.28@tcp --health 0 [ 1368.764630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1368.927110] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.28@tcp [ 1369.965332] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1369.971860] LNet: Skipped 1 previous similar message [ 1384.082577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1384.210414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1384.233984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1384.252481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.28@tcp --health 500 [ 1384.329606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1390.907890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1390.963878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1390.978612] LNet: 67647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1390.984380] LNetError: 67647:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.990062] LNetError: 67647:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1390.995293] LNet: Removed LNI 192.168.202.28@tcp [ 1391.003192] LNet: Removed LNI 192.168.202.28@tcp1 [ 1391.015293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1391.058909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1391.083454] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1391.586970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1391.601792] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1392.273625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1392.308924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1392.376773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1392.412873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1392.444746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1392.475361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1392.505922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1392.542589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.28@tcp --health 0 [ 1393.120137] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1393.124320] LNet: Skipped 3 previous similar messages [ 1393.861810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1404.121021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1404.151326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1404.181703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1410.624732] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 18:12:16 (1786399936) [ 1411.399524] LNet: 68692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.412099] LNetError: 68692:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.418394] LNetError: 68692:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1411.428132] LNet: Removed LNI 192.168.202.28@tcp [ 1411.447200] LNet: Removed LNI 192.168.202.28@tcp1 [ 1412.029723] Key type .llcrypt unregistered [ 1412.032638] Key type ._llcrypt unregistered [ 1418.563060] Key type ._llcrypt registered [ 1418.568185] Key type .llcrypt registered [ 1418.629654] LNet: 68916:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1418.629654] mask usage: [+|-] ... [ 1418.815642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1418.835884] alg: No test for adler32 (adler32-zlib) [ 1419.736466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1419.752085] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1429.704160] Lustre: DEBUG MARKER: Initial discovery [ 1429.763537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.128@tcp [ 1432.686821] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1432.782667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1432.852846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.128@tcp [ 1432.896568] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1432.934406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1432.998211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.128@tcp -r 1 -m GET -l 3 [ 1433.055679] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1433.108330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.128@tcp -r 1 -m GET -e local_error [ 1433.190795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.128@tcp -r 1 -m PUT -l 6 [ 1435.193228] Lustre: DEBUG MARKER: Force oleg228-client.virtnet to discover 192.168.202.128@tcp (in background) [ 1435.258125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.128@tcp [ 1436.417336] Lustre: DEBUG MARKER: Force oleg228-server to discover 192.168.202.28@tcp [ 1438.370828] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1439.129868] Lustre: DEBUG MARKER: Wait for 69447 [ 1440.502674] Lustre: DEBUG MARKER: Finished wait on 69447 [ 1440.680612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1440.756631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1440.758775] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1441.042579] LNet: 69907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1441.045927] LNetError: 69907:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1441.060807] LNet: Removed LNI 192.168.202.28@tcp [ 1441.438364] Key type .llcrypt unregistered [ 1441.439950] Key type ._llcrypt unregistered [ 1457.769695] Key type ._llcrypt registered [ 1457.773353] Key type .llcrypt registered [ 1459.128245] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 18:13:05 (1786399985) [ 1459.840450] Key type .llcrypt unregistered [ 1459.842774] Key type ._llcrypt unregistered [ 1467.286457] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1467.513264] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1467.520151] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1467.529825] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1470.333868] Key type ._llcrypt registered [ 1470.345181] Key type .llcrypt registered [ 1470.407459] LNet: 70950:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1470.407459] mask usage: [+|-] ... [ 1470.668878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1470.680698] alg: No test for adler32 (adler32-zlib) [ 1471.591329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.685277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1471.756105] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1472.498909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1472.517546] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1480.356642] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 18:13:26 (1786400006) [ 1480.682153] LNet: 71984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.698760] LNetError: 71984:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1480.705250] LNet: Removed LNI 10.1.2.3@tcp [ 1480.714482] LNet: Removed LNI 192.168.202.28@tcp [ 1481.499702] Key type .llcrypt unregistered [ 1481.502597] Key type ._llcrypt unregistered [ 1489.182711] Key type ._llcrypt registered [ 1489.196367] Key type .llcrypt registered [ 1489.261320] LNet: 72233:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1489.261320] mask usage: [+|-] ... [ 1489.531580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1489.566969] alg: No test for adler32 (adler32-zlib) [ 1490.497689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1490.579687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1490.624329] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1491.558631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1491.577873] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1493.066304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.28@tcp [ 1494.112495] LNetError: Unexpected error -22 connecting to 192.168.202.28@tcp at host 192.168.202.28:988 [ 1501.008788] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 18:13:46 (1786400026) [ 1501.436114] LNet: 73116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.454871] LNetError: 73116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1501.460604] LNet: Removed LNI 10.1.2.3@tcp [ 1501.481382] LNet: Removed LNI 192.168.202.28@tcp [ 1502.236989] Key type .llcrypt unregistered [ 1502.242603] Key type ._llcrypt unregistered [ 1508.577352] Key type ._llcrypt registered [ 1508.580823] Key type .llcrypt registered [ 1508.611770] LNet: 73346:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1508.611770] mask usage: [+|-] ... [ 1508.783545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1508.821653] alg: No test for adler32 (adler32-zlib) [ 1509.834747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1509.889840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1509.914927] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1510.928659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1510.973522] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 1512.088564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.28@tcp1 --nid 192.168.202.28@tcp2 [ 1518.023712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp1 192.168.202.28@tcp2 [ 1518.050704] LNetError: 73950:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.28@tcp1 to 192.168.202.28@tcp2 - Route table may be misconfigured [ 1518.085996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp2 192.168.202.28@tcp1 [ 1526.026664] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 18:14:11 (1786400051) [ 1526.224199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1526.278528] LNet: 74516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.291384] LNetError: 74516:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.305267] LNet: Removed LNI 192.168.202.28@tcp1 [ 1527.334694] LNet: Removed LNI 192.168.202.28@tcp2 [ 1527.398363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.521039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1527.599806] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1528.735691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1528.769818] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1530.130336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 1530.191244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.28@tcp [ 1530.234174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp1 192.168.202.28@tcp1 [ 1530.320442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.28@tcp -d 192.168.202.28@tcp -e network_timeout [ 1530.369424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -e network_timeout [ 1530.452689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -e network_timeout [ 1530.531231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -e network_timeout [ 1530.589467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1530.612814] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1530.628962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1530.739193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1530.850763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1531.551175] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1537.959845] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 18:14:24 (1786400064) [ 1538.253824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1538.279686] LNet: 75386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1538.283693] LNetError: 75386:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1538.295169] LNetError: 75386:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1538.301483] LNet: Removed LNI 192.168.202.28@tcp [ 1539.365434] LNet: Removed LNI 192.168.202.28@tcp1 [ 1539.412791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.535592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1539.842267] LNet: 75423:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.220476] Key type .llcrypt unregistered [ 1540.222783] Key type ._llcrypt unregistered [ 1552.167652] Key type ._llcrypt registered [ 1552.171772] Key type .llcrypt registered [ 1553.550439] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 18:14:39 (1786400079) [ 1554.117225] LNet: 76194:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1554.117225] mask usage: [+|-] ... [ 1554.285679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1554.300686] alg: No test for adler32 (adler32-zlib) [ 1555.329318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.162424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1558.962924] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 18:14:45 (1786400085) [ 1559.227064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.276109] LNet: 76521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.300607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.375765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1559.403558] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1560.320270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1560.341785] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1561.201663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1561.219371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp1 [ 1561.246509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp1 [ 1567.812633] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 18:14:53 (1786400093) [ 1569.576864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg228-server [ 1571.550247] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 18:14:57 (1786400097) [ 1573.899700] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg228-server [ 1575.796987] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 18:15:01 (1786400101) [ 1577.719084] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg228-server [ 1579.384838] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 18:15:05 (1786400105) [ 1581.623678] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg228-server [ 1583.416327] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 18:15:09 (1786400109) [ 1585.883781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg228-server [ 1587.800363] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 18:15:13 (1786400113) [ 1589.784563] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg228-server [ 1591.539766] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 18:15:17 (1786400117) [ 1593.692369] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg228-server [ 1595.764452] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 18:15:21 (1786400121) [ 1597.672676] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg228-server [ 1599.375105] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 18:15:25 (1786400125) [ 1601.197895] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg228-server [ 1603.733841] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 18:15:29 (1786400129) [ 1603.823616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.854947] LNet: 80392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1603.869522] LNetError: 80392:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.879088] LNet: Removed LNI 192.168.202.28@tcp [ 1604.904118] LNet: Removed LNI 192.168.202.28@tcp1 [ 1604.969443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.040535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1605.066384] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1605.814410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1605.873941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1606.045111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1606.068970] LNet: 80546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1606.077861] LNetError: 80546:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1606.082678] LNetError: 80546:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1607.155351] LNet: Removed LNI 192.168.202.28@tcp [ 1607.208418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.306036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1607.360757] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1608.501938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1608.644877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1608.688177] LNet: 80691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.695177] LNetError: 80691:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1609.768261] LNet: Removed LNI 192.168.202.28@tcp [ 1609.788843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.875516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.907058] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1610.839958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1610.958147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1611.275247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.306788] LNetError: 80838:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.325574] LNet: Removed LNI 192.168.202.28@tcp [ 1612.391857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.466270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1612.516934] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1613.411615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1613.713381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.730481] LNet: 80984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1613.738665] LNet: 80984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1614.762940] LNet: Removed LNI 192.168.202.28@tcp [ 1614.809658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.871664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1614.907341] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1615.762707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1615.900378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1616.341619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1616.391130] LNetError: 81131:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.404489] LNetError: 81131:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1616.440090] LNet: Removed LNI 192.168.202.28@tcp [ 1616.488933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.641398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1616.707737] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1618.189513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1618.406975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.498388] LNet: Removed LNI 192.168.202.28@tcp [ 1619.540680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.642849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1619.697254] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1621.053189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1621.177543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1621.560716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.633505] LNet: Removed LNI 192.168.202.28@tcp [ 1622.732851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.840971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1622.897323] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1624.258956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1624.557769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.587847] LNet: 81569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.591995] LNet: 81569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1624.597178] LNetError: 81569:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1624.600125] LNetError: 81569:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1625.645979] LNet: Removed LNI 192.168.202.28@tcp [ 1625.728722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.827641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1625.860874] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1626.868351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1626.973806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1628.546687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1629.611666] LNet: Removed LNI 192.168.202.28@tcp [ 1629.643572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.776408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1629.837942] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1631.496254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1631.891646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1632.942918] LNet: Removed LNI 192.168.202.28@tcp [ 1633.009108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.176714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1633.228869] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1634.227915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1634.377447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1634.748961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.815195] LNet: Removed LNI 192.168.202.28@tcp [ 1635.856737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.951133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1635.980608] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1637.429865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1637.868836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.959419] LNet: Removed LNI 192.168.202.28@tcp [ 1638.994678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.100572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1639.179749] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1640.181905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1640.267572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1640.311861] LNet: Removed LNI 192.168.202.28@tcp [ 1640.334140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.404670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1640.454124] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1641.446625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1647.659715] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 18:16:14 (1786400174) [ 1647.888903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.904501] LNet: 83000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1647.908669] LNet: 83000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1647.913316] LNetError: 83000:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1647.918701] LNetError: 83000:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1647.929676] LNet: Removed LNI 192.168.202.28@tcp [ 1647.948073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.984318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1648.000918] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 1648.970770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.994311] LNet: Removed LNI 192.168.202.28@tcp231 [ 1649.019823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.118199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1649.138460] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 1649.141538] LNet: 83147:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1649.667124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1649.701080] LNet: Removed LNI 192.168.202.28@tcp231 [ 1649.716447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1649.745990] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 1650.842414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1650.871926] LNet: Removed LNI 192.168.202.28@tcp231 [ 1650.900561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.992887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1651.019441] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/0] [ 1651.024027] LNet: 83299:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1651.546082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1651.578779] LNet: Removed LNI 192.168.202.28@tcp231 [ 1651.605285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1651.627558] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/0] [ 1652.778950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1652.808622] LNet: Removed LNI 192.168.202.28@tcp231 [ 1652.828557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.876921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1652.894370] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/60] [ 1652.898462] LNet: 83450:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1653.339239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1653.370778] LNet: Removed LNI 192.168.202.28@tcp231 [ 1653.383716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1653.422976] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/60] [ 1654.775903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.842934] LNet: Removed LNI 192.168.202.28@tcp231 [ 1654.903947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.032144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1655.080304] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 1655.086904] LNet: 83602:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1655.716720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1655.750299] LNet: Removed LNI 192.168.202.28@tcp231 [ 1655.767431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1655.817033] LNet: Added LNI 192.168.202.28@tcp231 [8/256/0/180] [ 1665.485615] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 18:16:30 (1786400190) [ 1665.759300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1665.792189] LNet: Removed LNI 192.168.202.28@tcp231 [ 1665.812805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.883495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1665.917423] LNet: Added LNI 192.168.202.28@tcp232 [8/256/0/180] [ 1666.600517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1666.657327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1666.732285] LNet: Removed LNI 192.168.202.28@tcp232 [ 1666.758487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1666.821198] LNet: Added LNI 192.168.202.28@tcp232 [8/256/0/180] [ 1667.467950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1667.497514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1667.525515] LNet: Removed LNI 192.168.202.28@tcp232 [ 1667.550452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1667.577674] LNet: Added LNI 192.168.202.28@tcp232 [8/256/0/180] [ 1668.514759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1668.569818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1668.617571] LNet: Removed LNI 192.168.202.28@tcp232 [ 1668.630325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1668.660578] LNet: Added LNI 192.168.202.28@tcp232 [8/256/0/180] [ 1669.490854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1669.537880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1669.579638] LNet: Removed LNI 192.168.202.28@tcp232 [ 1669.606296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1676.431992] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 18:16:42 (1786400202) [ 1677.825648] Key type .llcrypt unregistered [ 1677.827825] Key type ._llcrypt unregistered [ 1685.779658] Key type ._llcrypt registered [ 1685.780818] Key type .llcrypt registered [ 1685.819389] LNet: 85670:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1685.819389] mask usage: [+|-] ... [ 1686.067218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1686.089746] alg: No test for adler32 (adler32-zlib) [ 1687.041778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1687.072475] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1694.552611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1703.800616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.854622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1703.880173] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1705.668326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_interrupt -r 2 [ 1705.746289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_interrupt -r 2 [ 1705.820230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_interrupt -r 2 [ 1705.872487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_interrupt -r 2 [ 1705.950325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_interrupt -r 2 [ 1705.989144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_interrupt -r 2 [ 1706.048673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_interrupt -r 2 [ 1706.094956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_interrupt -r 2 [ 1706.131904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1706.170926] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1706.175770] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1706.444650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1706.481085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1706.511459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1706.546585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1706.585263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1706.637633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1706.685258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1706.744181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1706.758205] LNet: 86103:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1706.782388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1706.817578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1706.885042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1706.934257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1706.974979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1707.014597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1707.520568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_dropped -r 2 [ 1707.551628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_dropped -r 2 [ 1707.580204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_dropped -r 2 [ 1707.617190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_dropped -r 2 [ 1707.646887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_dropped -r 2 [ 1707.665329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_dropped -r 2 [ 1707.696287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_dropped -r 2 [ 1707.726636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_dropped -r 2 [ 1707.748475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1707.767804] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1707.969397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1708.000485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1708.027534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1708.061397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1708.094371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1708.140261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1708.180059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1708.206912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1708.221838] LNet: 86223:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1708.242493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1708.287502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1708.320915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1708.358502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1708.385917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1708.419260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1708.878303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_aborted -r 2 [ 1708.910542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_aborted -r 2 [ 1708.936962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_aborted -r 2 [ 1708.999231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_aborted -r 2 [ 1709.039237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_aborted -r 2 [ 1709.082466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_aborted -r 2 [ 1709.155216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_aborted -r 2 [ 1709.203472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_aborted -r 2 [ 1709.267221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1709.290937] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1709.531350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1709.578448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1709.616853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1709.638398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1709.680644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1709.722253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1709.770369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1709.826479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1709.853619] LNet: 86343:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1709.890223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1709.939801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1709.979471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1710.047904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1710.102313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1710.135210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1710.610672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_no_route -r 2 [ 1710.630168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_no_route -r 2 [ 1710.657521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_no_route -r 2 [ 1710.704921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_no_route -r 2 [ 1710.749988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_no_route -r 2 [ 1710.779869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_no_route -r 2 [ 1710.823526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_no_route -r 2 [ 1710.855559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_no_route -r 2 [ 1710.918638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1711.295336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1711.335903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1711.368843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1711.415162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1711.482448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1711.524607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1711.552748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1711.583876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1711.615373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1711.644518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1711.685164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1711.766096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1711.823655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1711.862056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1712.507801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e local_timeout -r 2 [ 1712.590280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e local_timeout -r 2 [ 1712.661344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e local_timeout -r 2 [ 1712.718606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e local_timeout -r 2 [ 1712.801610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e local_timeout -r 2 [ 1712.858733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e local_timeout -r 2 [ 1712.915746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e local_timeout -r 2 [ 1712.969428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e local_timeout -r 2 [ 1713.050431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1713.125386] LNet: There was an unexpected network error while writing to 192.168.202.128: rc = -22 [ 1713.129629] LNet: Skipped 1 previous similar message [ 1713.495555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1713.517270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1713.609037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1713.703734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1713.760201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1713.836448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1713.904569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1713.954719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1713.986879] LNet: 86584:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1714.006936] LNet: 86584:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1714.031399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1714.077544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1714.146080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1714.191051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1714.251593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1714.347212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1714.976248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp -m GET -e remote_dropped -r 2 [ 1715.019508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.128@tcp1 -m GET -e remote_dropped -r 2 [ 1715.066325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp -m GET -e remote_dropped -r 2 [ 1715.172472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.28@tcp -d 192.168.202.28@tcp1 -m GET -e remote_dropped -r 2 [ 1715.227859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp -m GET -e remote_dropped -r 2 [ 1715.299534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.128@tcp1 -m GET -e remote_dropped -r 2 [ 1715.347493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp -m GET -e remote_dropped -r 2 [ 1715.393345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.28@tcp1 -d 192.168.202.28@tcp1 -m GET -e remote_dropped -r 2 [ 1715.478248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.28@tcp 192.168.202.128@tcp [ 1715.521194] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp [ 1715.842940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1715.868517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1715.928343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1715.965890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1716.038545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1716.068813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1716.122797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1716.177507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1716.225864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1716.280212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1716.343804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1716.396824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1716.460912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1716.562808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1723.734508] LNet: 86779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.747415] LNetError: 86779:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.768266] LNet: Removed LNI 192.168.202.28@tcp [ 1724.772474] LNetError: 86779:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.789483] LNet: Removed LNI 192.168.202.28@tcp1 [ 1725.152590] Key type .llcrypt unregistered [ 1725.156639] Key type ._llcrypt unregistered [ 1745.774489] Key type ._llcrypt registered [ 1745.778510] Key type .llcrypt registered [ 1747.120657] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 18:17:52 (1786400272) [ 1747.600916] LNet: 87586:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1747.600916] mask usage: [+|-] ... [ 1747.774809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1747.790956] alg: No test for adler32 (adler32-zlib) [ 1748.743945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.827966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1748.884526] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1749.720984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1749.768072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.28@tcp [ 1749.790258] LNetError: 87759:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.28@tcp to recovery [ 1749.807686] LNet: 1 local NIs in recovery (showing 1): 192.168.202.28@tcp [ 1749.876709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.28@tcp [ 1749.956183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1757.775512] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 18:18:03 (1786400283) [ 1757.847218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.872703] LNet: 88342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.881016] LNetError: 88342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.915072] LNet: Removed LNI 192.168.202.28@tcp [ 1757.952696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.026413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1758.043985] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1758.565938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1758.583100] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 1759.349755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.28@tcp [ 1759.389719] LNetError: 88614:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.28@tcp to recovery [ 1760.033523] LNet: There was an unexpected network error while writing to 192.168.202.28: rc = -22 [ 1760.769522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.28@tcp [ 1770.488937] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 18:18:16 (1786400296) [ 1772.436854] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg228-server [ 1774.292278] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 18:18:20 (1786400300) [ 1776.906691] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg228-server [ 1779.465812] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 18:18:24 (1786400304) [ 1781.692953] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg228-server [ 1783.703623] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 18:18:29 (1786400309) [ 1783.843582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.896929] LNet: 90239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.902348] LNetError: 90239:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.909691] LNet: Removed LNI 192.168.202.28@tcp [ 1783.931172] LNet: Removed LNI 192.168.202.28@tcp1 [ 1784.000707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.085657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1784.114788] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1784.780354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1784.848597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1784.893295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1784.961838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1785.013564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1785.061374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1792.252833] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 18:18:37 (1786400317) [ 1792.415643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1792.466538] LNet: 90965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1792.475283] LNetError: 90965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1792.480692] LNetError: 90965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1792.501988] LNet: Removed LNI 192.168.202.28@tcp [ 1792.517526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.603529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1792.621271] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1800.820812] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 18:18:46 (1786400326) [ 1801.120485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.154486] LNet: 91674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.169709] LNetError: 91674:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.185468] LNet: Removed LNI 192.168.202.28@tcp [ 1801.197257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.244771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1801.265447] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1808.652431] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 18:18:54 (1786400334) [ 1810.341307] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1812.447421] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 18:18:58 (1786400338) [ 1812.987455] LNet: 92710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.996456] LNetError: 92710:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.021292] LNet: Removed LNI 192.168.202.28@tcp [ 1813.759339] Key type .llcrypt unregistered [ 1813.761066] Key type ._llcrypt unregistered [ 1821.394313] Key type ._llcrypt registered [ 1821.396300] Key type .llcrypt registered [ 1821.450487] LNet: 92939:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1821.450487] mask usage: [+|-] ... [ 1821.633572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1821.647987] alg: No test for adler32 (adler32-zlib) [ 1822.569324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1822.603585] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1830.408605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 1835.453526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.128@tcp [ 1835.950745] LNet: 93127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.955826] LNetError: 93127:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.982704] LNet: Removed LNI 192.168.202.28@tcp [ 1836.452577] Key type .llcrypt unregistered [ 1836.457822] Key type ._llcrypt unregistered [ 1848.543290] Key type ._llcrypt registered [ 1848.549479] Key type .llcrypt registered [ 1849.839585] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1851.156567] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1853.370563] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 18:19:39 (1786400379) [ 1855.466843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1857.386523] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 18:19:43 (1786400383) [ 1859.922772] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg228-server [ 1861.459845] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 18:19:47 (1786400387) [ 1863.384623] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg228-server [ 1865.422157] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 18:19:51 (1786400391) [ 1867.043531] LNet: 95177:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1867.043531] mask usage: [+|-] ... [ 1867.809942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1867.862092] alg: No test for adler32 (adler32-zlib) [ 1868.889851] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 1869.110687] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1870.863180] Key type lgssc registered [ 1872.654792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1899.607331] Key type lgssc unregistered [ 1899.880568] LNet: 96304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1899.889454] LNetError: 96304:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1899.909929] LNet: Removed LNI 192.168.202.28@tcp [ 1900.771396] Key type .llcrypt unregistered [ 1900.774814] Key type ._llcrypt unregistered [ 1915.325276] Key type ._llcrypt registered [ 1915.327064] Key type .llcrypt registered [ 1916.830951] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 18:20:42 (1786400442) [ 1917.093126] LNet: 97100:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1917.093126] mask usage: [+|-] ... [ 1917.251343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1917.262591] alg: No test for adler32 (adler32-zlib) [ 1918.199553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.301805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1918.376536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1918.427211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1918.491523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1925.790099] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 18:20:51 (1786400451) [ 1926.224187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1926.242691] LNet: 97718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1926.255436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1926.292522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1926.307890] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1927.386949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1927.402361] LNet: 97862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1927.407205] LNetError: 97862:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1927.415824] LNet: Removed LNI 192.168.202.28@tcp [ 1927.432714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1927.511506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1927.550560] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1928.141941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1928.159895] LNetError: 97890:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.173812] LNet: Removed LNI 192.168.202.28@tcp [ 1928.186273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1928.223419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1928.243217] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1935.753704] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 18:21:01 (1786400461) [ 1935.940809] LNet: 98486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1935.948088] LNet: 98486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1935.954990] LNetError: 98486:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1935.985880] LNet: Removed LNI 192.168.202.28@tcp [ 1936.772310] Key type .llcrypt unregistered [ 1936.774780] Key type ._llcrypt unregistered [ 1944.050951] Key type ._llcrypt registered [ 1944.052595] Key type .llcrypt registered [ 1944.113925] LNet: 98722:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1944.113925] mask usage: [+|-] ... [ 1944.344036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1944.374961] alg: No test for adler32 (adler32-zlib) [ 1945.367314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1945.500778] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 1945.508230] LNet: *** cfs_fail_loc=e003, val=100*** [ 1946.442664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 1946.483822] LNet: *** cfs_fail_loc=e003, val=100*** [ 1946.488740] LNet: Skipped 2 previous similar messages [ 1946.501235] Ping reply received with mapping: [ 1946.505350] found metadata reply NI[192.168.202.28@tcp] [ 1946.511252] 1c ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1946.514935] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1946.519655] 5a 5a 5a 5a 5a 5a 5a 5a [ 1946.783748] LNet: 98888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1946.799239] LNetError: 98888:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1946.831284] LNet: Removed LNI 192.168.202.28@tcp [ 1947.698330] Key type .llcrypt unregistered [ 1947.700536] Key type ._llcrypt unregistered [ 1959.458247] Key type ._llcrypt registered [ 1959.460128] Key type .llcrypt registered [ 1960.779662] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 18:21:26 (1786400486) [ 1961.491760] Key type .llcrypt unregistered [ 1961.493325] Key type ._llcrypt unregistered [ 1968.639684] Key type ._llcrypt registered [ 1968.641559] Key type .llcrypt registered [ 1968.698654] LNet: 99884:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1968.698654] mask usage: [+|-] ... [ 1968.910478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1968.928499] alg: No test for adler32 (adler32-zlib) [ 1970.177166] LNetError: 99934:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1970.634864] Key type .llcrypt unregistered [ 1970.638353] Key type ._llcrypt unregistered [ 1975.844825] Key type ._llcrypt registered [ 1975.848838] Key type .llcrypt registered [ 1977.365413] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 18:21:43 (1786400503) [ 1977.959263] Key type .llcrypt unregistered [ 1977.960519] Key type ._llcrypt unregistered [ 1985.087761] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1985.098884] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1987.810635] Key type ._llcrypt registered [ 1987.826172] Key type .llcrypt registered [ 1987.879594] LNet: 100802:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1987.879594] mask usage: [+|-] ... [ 1988.058923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1988.077790] alg: No test for adler32 (adler32-zlib) [ 1988.992714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.058479] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2000.782790] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 18:22:06 (1786400526) [ 2001.126968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2001.146649] LNet: 101714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2001.156674] LNetError: 101714:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2001.166367] LNet: Removed LNI 10.1.2.3@tcp [ 2001.187696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.002363] Key type .llcrypt unregistered [ 2002.005572] Key type ._llcrypt unregistered [ 2010.079940] Key type ._llcrypt registered [ 2010.088946] Key type .llcrypt registered [ 2010.198417] LNet: 101946:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2010.198417] mask usage: [+|-] ... [ 2010.449585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2010.497979] alg: No test for adler32 (adler32-zlib) [ 2011.609139] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2011.718193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2011.889525] LNet: 102003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.895512] LNetError: 102003:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2011.915453] LNet: Removed LNI 192.168.202.28@tcp [ 2012.820559] Key type .llcrypt unregistered [ 2012.822902] Key type ._llcrypt unregistered [ 2026.862822] Key type ._llcrypt registered [ 2026.867381] Key type .llcrypt registered [ 2028.742744] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 18:22:34 (1786400554) [ 2029.381269] Key type .llcrypt unregistered [ 2029.382586] Key type ._llcrypt unregistered [ 2036.319375] Key type ._llcrypt registered [ 2036.322207] Key type .llcrypt registered [ 2036.410633] LNet: 102988:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2036.410633] mask usage: [+|-] ... [ 2036.606957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2036.626297] alg: No test for adler32 (adler32-zlib) [ 2045.149312] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 18:22:50 (1786400570) [ 2045.592208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2045.652328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2045.765223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2045.826752] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2046.630516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2046.661214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2046.690423] LNet: Removed LNI 192.168.202.28@tcp [ 2046.953329] LNet: 103747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2047.482200] Key type .llcrypt unregistered [ 2047.484719] Key type ._llcrypt unregistered [ 2061.669478] Key type ._llcrypt registered [ 2061.680284] Key type .llcrypt registered [ 2063.529896] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 18:23:08 (1786400588) [ 2063.805975] LNet: 104525:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2063.805975] mask usage: [+|-] ... [ 2063.947078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2063.963404] alg: No test for adler32 (adler32-zlib) [ 2064.935420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2064.981695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2065.014850] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2065.973830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2066.032752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.28@tcp [ 2066.061237] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2066.067190] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2066.071508] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2066, 0, 0 [ 2066.077903] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2066.083051] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2066.090448] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2066.094992] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2066, 0, 0 [ 2066.100786] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2066.104375] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2066.108783] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2066.113527] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2066, 0, 2116 [ 2066.121935] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2066.126523] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2066.132776] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2066.139732] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2066, 0, 2116 [ 2066.143149] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2073.669439] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 18:23:19 (1786400599) [ 2074.397984] LNet: 105299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2074.409505] LNetError: 105299:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2074.434962] LNet: Removed LNI 192.168.202.28@tcp [ 2075.361381] Key type .llcrypt unregistered [ 2075.366899] Key type ._llcrypt unregistered [ 2083.983323] Key type ._llcrypt registered [ 2083.986526] Key type .llcrypt registered [ 2084.026412] LNet: 105523:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2084.026412] mask usage: [+|-] ... [ 2084.131937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2084.140628] alg: No test for adler32 (adler32-zlib) [ 2085.189975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2085.248916] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2094.163852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 2105.300700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.381613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2105.412732] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 2106.541157] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2106.898261] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2106.905561] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2106.919305] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2109.571413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2109.604264] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2124.780048] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 18:24:09 (1786400649) [ 2124.948053] LNet: 106801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2124.964864] LNetError: 106801:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2124.978304] LNet: Removed LNI 192.168.202.28@tcp [ 2126.047641] LNet: Removed LNI 192.168.202.28@tcp1 [ 2126.052136] LNetError: 106801:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2126.059928] LNetError: 106801:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2126.076542] LNet: Removed LNI 10.1.2.3@tcp99 [ 2127.224390] Key type .llcrypt unregistered [ 2127.230624] Key type ._llcrypt unregistered [ 2134.893347] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2134.903898] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2137.745529] Key type ._llcrypt registered [ 2137.755224] Key type .llcrypt registered [ 2137.842263] LNet: 107076:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2137.842263] mask usage: [+|-] ... [ 2138.172207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2138.220292] alg: No test for adler32 (adler32-zlib) [ 2139.273890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.371795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2139.415782] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2140.648363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2140.679986] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2144.639442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2144.711494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2144.783859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2144.806322] LNetError: 107555:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2144.871644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2144.938889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2145.302663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2145.373658] LNet: 107575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2145.382574] LNetError: 107575:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2145.388471] LNet: Removed LNI 10.1.2.3@tcp [ 2146.409422] LNet: Removed LNI 192.168.202.28@tcp [ 2147.288393] Key type .llcrypt unregistered [ 2147.291397] Key type ._llcrypt unregistered [ 2160.600276] Key type ._llcrypt registered [ 2160.602022] Key type .llcrypt registered [ 2161.784538] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 18:24:47 (1786400687) [ 2162.372426] LNet: 108375:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2162.372426] mask usage: [+|-] ... [ 2162.613458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2162.641422] alg: No test for adler32 (adler32-zlib) [ 2163.643525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2163.730460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2163.771789] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2164.654843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.28@tcp [ 2164.748958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg228-client@tcp [ 2172.296591] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 18:24:58 (1786400698) [ 2172.781647] LNet: 109111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.796099] LNetError: 109111:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2172.812434] LNet: Removed LNI 192.168.202.28@tcp [ 2173.760568] Key type .llcrypt unregistered [ 2173.763084] Key type ._llcrypt unregistered [ 2180.859384] Key type ._llcrypt registered [ 2180.876625] Key type .llcrypt registered [ 2181.054264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2181.074824] alg: No test for adler32 (adler32-zlib) [ 2181.985454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2182.019141] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2182.054757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2189.032491] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 18:25:14 (1786400714) [ 2190.748369] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2192.707285] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 18:25:18 (1786400718) [ 2194.552119] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2196.800151] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 18:25:22 (1786400722) [ 2196.931984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2196.987800] LNet: 110524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.998945] LNetError: 110524:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2197.021229] LNet: Removed LNI 192.168.202.28@tcp [ 2197.087561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2197.162069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2197.184556] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2198.501324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2198.548910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2206.681837] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 18:25:32 (1786400732) [ 2206.852888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2206.886870] LNet: 111235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.892864] LNetError: 111235:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2206.910189] LNet: Removed LNI 192.168.202.28@tcp [ 2206.947573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2207.090482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2207.137768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2207.886578] Key type .llcrypt unregistered [ 2207.889273] Key type ._llcrypt unregistered [ 2221.928596] Key type ._llcrypt registered [ 2221.934335] Key type .llcrypt registered [ 2223.399172] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 18:25:49 (1786400749) [ 2223.874375] LNet: 112049:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2223.874375] mask usage: [+|-] ... [ 2224.054133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2224.078602] alg: No test for adler32 (adler32-zlib) [ 2225.004672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.121422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2225.143285] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2226.056505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2226.156592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2233.837810] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 18:25:59 (1786400759) [ 2233.945556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.005534] LNet: 112792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.012659] LNetError: 112792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2234.035358] LNet: Removed LNI 192.168.202.28@tcp [ 2234.064380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2234.137436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2234.186373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2241.382929] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 18:26:07 (1786400767) [ 2241.571405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2241.595589] LNet: 113376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2241.626371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.744504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2242.346452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2242.408565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2242.554353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2243.101885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2243.163776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2243.244614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2243.265329] LNet: 113413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2243.283430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.372720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2243.764592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2243.790851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2243.836851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2244.410807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2244.438115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2251.751533] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 18:26:17 (1786400777) [ 2252.067426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2252.119420] LNet: 114013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.169903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2252.309780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2252.366312] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2253.475669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2253.513444] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2254.638294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2254.711541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2255.053646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2255.092064] LNetError: 114293:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2255.096728] LNet: Removed LNI 192.168.202.28@tcp [ 2255.106549] LNet: Removed LNI 192.168.202.28@tcp2 [ 2255.122923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2255.184984] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2255.190443] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2255.206296] LNet: 114299:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2255.932386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2256.177231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2256.199762] LNet: 114336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2256.205925] LNet: 114336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2256.216229] LNetError: 114336:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2256.225941] LNetError: 114336:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2256.233936] LNet: Removed LNI 192.168.202.28@tcp [ 2256.250242] LNet: Removed LNI 192.168.202.28@tcp2 [ 2256.268667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.336160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2256.357144] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2257.166323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2257.189956] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2258.244759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2258.297962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2258.405691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2258.715139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2258.762468] LNetError: 114621:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.768315] LNetError: 114621:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2258.774712] LNet: Removed LNI 192.168.202.28@tcp [ 2259.820197] LNet: Removed LNI 192.168.202.28@tcp2 [ 2259.888318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2259.991793] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2260.001372] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2260.009682] LNet: 114627:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2261.185636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2261.455205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2261.475500] LNet: Removed LNI 192.168.202.28@tcp [ 2262.509223] LNet: Removed LNI 192.168.202.28@tcp2 [ 2262.555468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2262.661011] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2262.685763] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2263.718808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2263.746477] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2264.761862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2264.852943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2265.127621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2265.154954] LNet: 114944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2265.161944] LNet: 114944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2265.172192] LNetError: 114944:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2265.179611] LNetError: 114944:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2265.186532] LNet: Removed LNI 192.168.202.28@tcp [ 2265.196145] LNet: Removed LNI 192.168.202.28@tcp2 [ 2265.225828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2265.334991] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2265.343413] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2265.355992] LNet: 114950:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2266.143203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2266.420582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2267.487315] LNet: Removed LNI 192.168.202.28@tcp [ 2267.504793] LNet: Removed LNI 192.168.202.28@tcp2 [ 2267.539290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2267.623776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2267.650054] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2268.573403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2268.609037] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2269.619122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2269.674113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2269.762607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2270.006293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2271.075354] LNet: Removed LNI 192.168.202.28@tcp [ 2272.111142] LNet: Removed LNI 192.168.202.28@tcp2 [ 2272.223227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.315670] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2272.320457] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2272.327956] LNet: 115278:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2273.045446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2273.292803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2273.309265] LNetError: 115316:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.313431] LNetError: 115316:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2273.318653] LNet: Removed LNI 192.168.202.28@tcp [ 2274.342184] LNet: Removed LNI 192.168.202.28@tcp2 [ 2274.407952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2274.545708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2274.597162] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2275.297769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2275.316918] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2276.322765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2276.351859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2276.393947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2276.775827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2277.855240] LNet: Removed LNI 192.168.202.28@tcp [ 2277.874450] LNet: Removed LNI 192.168.202.28@tcp2 [ 2277.928746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2278.046060] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2278.051188] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2278.056745] LNet: 115619:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2278.453455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2278.896701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2279.967465] LNet: Removed LNI 192.168.202.28@tcp [ 2279.981181] LNet: Removed LNI 192.168.202.28@tcp2 [ 2280.008627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2280.143044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2280.197838] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2280.962892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2280.985037] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2281.875738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2281.912228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2281.945355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2282.035838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2282.510347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2282.537820] LNet: 115972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2282.542610] LNet: 115972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2283.551253] LNet: Removed LNI 192.168.202.28@tcp [ 2284.597176] LNet: Removed LNI 192.168.202.28@tcp2 [ 2284.673301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.812101] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2284.829379] LNet: Added LNI 192.168.202.28@tcp2 [8/256/0/180] [ 2285.529932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2293.877480] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2295.634660] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 18:27:01 (1786400821) [ 2296.063353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2296.110288] LNetError: 116738:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2296.116895] LNetError: 116738:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2296.125164] LNet: Removed LNI 192.168.202.28@tcp [ 2297.196768] LNet: Removed LNI 192.168.202.28@tcp2 [ 2297.288804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2297.852807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2297.889220] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2298.709843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2298.727487] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2299.868510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2299.910739] LNet: Removed LNI 10.1.2.3@tcp [ 2299.927892] LNet: Removed LNI 192.168.202.28@tcp [ 2299.962951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2300.095401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2300.127229] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2300.142500] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2300.684251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2300.723764] LNet: Removed LNI 192.168.202.28@tcp [ 2300.737180] LNet: Removed LNI 10.1.2.3@tcp [ 2308.664624] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 18:27:14 (1786400834) [ 2308.793406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2308.878242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2309.054594] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2309.332172] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2309.345156] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2309.351507] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2311.782516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2311.823522] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2313.006469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2313.036458] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2315.459575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.28@tcp [ 2315.501593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2315.580513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.28@tcp -m PUT -e local_timeout -r 1 [ 2315.694180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2315.820773] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2315.830782] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2316.178779] LNetError: 117676:0:(lib-move.c:4714:lnet_parse()) 192.168.202.28@tcp, src 192.168.202.28@tcp: Dropping PUT (error -108 looking up sender) [ 2317.221112] LNet: 118136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2317.232864] LNet: 118136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2318.303820] LNet: Removed LNI 10.1.2.3@tcp [ 2319.340188] LNet: Removed LNI 192.168.202.28@tcp [ 2319.855540] Key type .llcrypt unregistered [ 2319.859379] Key type ._llcrypt unregistered [ 2333.771740] Key type ._llcrypt registered [ 2333.775757] Key type .llcrypt registered [ 2335.212809] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 18:27:40 (1786400860) [ 2335.731566] LNet: 118907:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2335.731566] mask usage: [+|-] ... [ 2335.941197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2335.968474] alg: No test for adler32 (adler32-zlib) [ 2336.988328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.406123] LNet: 118989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.944372] Key type .llcrypt unregistered [ 2337.946963] Key type ._llcrypt unregistered [ 2344.640802] Key type ._llcrypt registered [ 2344.643035] Key type .llcrypt registered [ 2344.706183] LNet: 119218:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2344.706183] mask usage: [+|-] ... [ 2344.982656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2345.001719] alg: No test for adler32 (adler32-zlib) [ 2346.015574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2346.039825] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2353.423758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 2353.689494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2354.447648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.128@tcp [ 2354.738608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2354.771133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2354.797325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2354.823823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2354.863843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2354.899860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2354.937488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2354.993883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2355.023742] LNet: 119439:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2355.052190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2355.097394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2355.132385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2355.164066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2355.210551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2355.247078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2355.987853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2356.312052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2356.329106] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp [ 2356.365991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2356.411515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2356.450546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2356.497592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2356.532712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2356.588203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2356.646563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2356.665966] LNet: 119556:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2356.700453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2356.740198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2356.760822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2356.778629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2356.794587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2356.817716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2357.360869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.128@tcp [ 2357.706387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2357.751607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2357.810196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2357.846236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2357.886673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2357.936757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2358.003149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2358.067957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2358.111147] LNet: 119673:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2358.159140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2358.225218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2358.288954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2358.331157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2358.380960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2358.468423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2359.443483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2359.703669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2359.764573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2359.817249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2359.855720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2359.898904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2359.933046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2359.976442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2360.007589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2360.040228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2360.080370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2360.106945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2360.137965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2360.169932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2360.187929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2360.600751] LNet: 119837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2360.610484] LNetError: 119837:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2360.624260] LNet: Removed LNI 192.168.202.28@tcp [ 2361.119762] Key type .llcrypt unregistered [ 2361.121812] Key type ._llcrypt unregistered [ 2380.708603] Key type ._llcrypt registered [ 2380.721175] Key type .llcrypt registered [ 2382.623235] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 18:28:27 (1786400907) [ 2382.944812] LNet: 120643:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2382.944812] mask usage: [+|-] ... [ 2383.197422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2383.221246] alg: No test for adler32 (adler32-zlib) [ 2384.251533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2384.856754] LNet: 120725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2385.570721] Key type .llcrypt unregistered [ 2385.576260] Key type ._llcrypt unregistered [ 2393.454649] Key type ._llcrypt registered [ 2393.457247] Key type .llcrypt registered [ 2393.514903] LNet: 120954:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2393.514903] mask usage: [+|-] ... [ 2393.742881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2393.758593] alg: No test for adler32 (adler32-zlib) [ 2394.729250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2394.757593] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2403.491317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.128@tcp [ 2416.168787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2416.223534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2416.255217] LNet: Added LNI 192.168.202.28@tcp1 [8/256/0/180] [ 2417.299869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2418.064726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.128@tcp [ 2418.342116] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.128@tcp [ 2418.521873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2418.587447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2418.672726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2418.727174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2418.774479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2418.835616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2418.889375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2418.911787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2418.931775] LNet: 121379:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2418.953864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2418.979378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2419.005472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2419.033311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2419.062711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2419.094504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2419.790470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2420.201833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2420.244547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2420.294967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2420.363849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2420.412503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2420.463505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2420.533344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2420.581577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2420.604708] LNet: 121496:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2420.627551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2420.704137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2420.785099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2420.861865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2420.959737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2421.040869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2422.039388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.128@tcp [ 2422.380978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2422.431787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2422.506534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2422.597708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2422.658542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2422.698260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2422.725106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2422.832424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2422.855956] LNet: 121613:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2422.906335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2422.970531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2423.016895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2423.061453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2423.099622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2423.141994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2424.062468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2424.430568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2424.469393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2424.522359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2424.582356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2424.648398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2424.699596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2424.753766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2424.795448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2424.848476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2424.902688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2424.948635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2425.005793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2425.057189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2425.112384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2431.584300] LNet: 121812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2431.592272] LNetError: 121812:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2431.601915] LNet: Removed LNI 192.168.202.28@tcp [ 2431.618990] LNet: Removed LNI 192.168.202.28@tcp1 [ 2432.276685] Key type .llcrypt unregistered [ 2432.278413] Key type ._llcrypt unregistered [ 2451.608460] Key type ._llcrypt registered [ 2451.610057] Key type .llcrypt registered [ 2453.052487] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 18:29:38 (1786400978) [ 2462.779823] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 18:29:48 (1786400988) [ 2473.162145] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 18:29:58 (1786400998) [ 2483.194160] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 18:30:08 (1786401008) [ 2493.549807] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 18:30:19 (1786401019) [ 2501.856713] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 18:30:27 (1786401027) [ 2510.320287] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 18:30:36 (1786401036) [ 2519.595581] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 18:30:45 (1786401045) [ 2529.860731] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 18:30:55 (1786401055) [ 2539.026955] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 18:31:04 (1786401064) [ 2547.600769] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 18:31:13 (1786401073) [ 2557.127813] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 18:31:22 (1786401082) [ 2564.463027] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 18:31:30 (1786401090) [ 2574.028415] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 18:31:39 (1786401099) [ 2583.924516] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 18:31:49 (1786401109) [ 2592.626535] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 18:31:58 (1786401118) [ 2601.985951] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 18:32:07 (1786401127) [ 2611.644865] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 18:32:17 (1786401137) [ 2611.931854] LNet: 132355:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2611.931854] mask usage: [+|-] ... [ 2612.159743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2612.185743] alg: No test for adler32 (adler32-zlib) [ 2613.205825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2613.382955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2613.419294] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2614.996264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2615.025945] LNet: 132531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2615.030227] LNetError: 132531:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2615.042956] LNet: Removed LNI 192.168.202.28@tcp [ 2615.062268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2615.169424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2615.219393] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2623.493989] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 18:32:29 (1786401149) [ 2631.931743] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 18:32:37 (1786401157) [ 2640.461687] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 18:32:46 (1786401166) [ 2650.040482] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 18:32:55 (1786401175) [ 2658.180468] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 18:33:03 (1786401183) [ 2667.412583] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 18:33:13 (1786401193) [ 2675.676949] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 18:33:21 (1786401201) [ 2683.039189] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 18:33:28 (1786401208) [ 2690.232122] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 18:33:36 (1786401216) [ 2696.887851] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 18:33:42 (1786401222) [ 2705.867317] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 18:33:51 (1786401231) [ 2715.424795] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 18:34:00 (1786401240) [ 2724.903121] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2475 sec ========== 18:34:10 (1786401250) [ 2725.411310] LNet: 139997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2725.421880] LNetError: 139997:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2725.447631] LNet: Removed LNI 192.168.202.28@tcp [ 2726.303308] Key type .llcrypt unregistered [ 2726.305346] Key type ._llcrypt unregistered [ 2739.843405] Key type ._llcrypt registered [ 2739.846057] Key type .llcrypt registered [ 2739.912155] LNet: 140470:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2739.912155] mask usage: [+|-] ... [ 2740.064390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2740.091220] alg: No test for adler32 (adler32-zlib) [ 2741.155595] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 2741.331643] LNet: Added LNI 192.168.202.28@tcp [8/256/0/180] [ 2742.991339] Key type lgssc registered [ 2744.714381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2808.605088] Lustre: Mounted lustre-client [ 2808.765835] LNet: 141942:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2808.765835] mask usage: [+|-] ... [ 2814.186859] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2824.016563] Lustre: Unmounted lustre-client [ 2848.987951] Key type lgssc unregistered [ 2849.277183] LNet: 142924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2849.282598] LNetError: 142924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2849.300301] LNet: Removed LNI 192.168.202.28@tcp [ 2850.120451] Key type .llcrypt unregistered [ 2850.126049] Key type ._llcrypt unregistered