[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 425305804 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003123] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008375] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010023] pid_max: default: 32768 minimum: 301 [ 0.011133] LSM: Security Framework initializing [ 0.012058] Yama: becoming mindful. [ 0.013036] SELinux: Initializing. [ 0.015075] *** VALIDATE selinux *** [ 0.024177] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029432] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032122] *** VALIDATE tmpfs *** [ 0.033426] *** VALIDATE proc *** [ 0.035208] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.037271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041030] Spectre V2 : User space: Vulnerable [ 0.042013] Speculative Store Bypass: Vulnerable [ 0.044716] debug: unmapping init [mem 0xffffffffafe59000-0xffffffffafe60fff] [ 0.047123] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048704] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049023] ... version: 2 [ 0.050012] ... bit width: 48 [ 0.051015] ... generic registers: 4 [ 0.052014] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054017] ... fixed-purpose events: 3 [ 0.055015] ... event mask: 000000070000000f [ 0.057224] rcu: Hierarchical SRCU implementation. [ 0.059448] smp: Bringing up secondary CPUs ... [ 0.060644] x86: Booting SMP configuration: [ 0.061030] .... node #0, CPUs: #1 #2 #3 [ 0.065108] smp: Brought up 1 node, 4 CPUs [ 0.067019] smpboot: Max logical packages: 1 [ 0.068016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.227000] node 0 deferred pages initialised in 157ms [ 0.231103] devtmpfs: initialized [ 0.232340] x86/mm: Memory block size: 128MB [ 0.235985] gcov: version magic: 0x41383552 [ 0.239356] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.244135] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.246434] pinctrl core: initialized pinctrl subsystem [ 0.249262] [ 0.249873] ************************************************************* [ 0.252017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.254013] ** ** [ 0.257018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.259015] ** ** [ 0.262016] ** This means that this kernel is built to expose internal ** [ 0.264015] ** IOMMU data structures, which may compromise security on ** [ 0.266013] ** your system. ** [ 0.268015] ** ** [ 0.271015] ** If you see this message and you are not debugging the ** [ 0.273016] ** kernel, report this immediately to your vendor! ** [ 0.275014] ** ** [ 0.278020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.280014] ************************************************************* [ 0.282725] NET: Registered protocol family 16 [ 0.285599] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.288081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.291087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.294227] cpuidle: using governor menu [ 0.296797] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.299698] PCI: Using configuration type 1 for base access [ 0.302130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.313070] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.314021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.316059] cryptd: max_cpu_qlen set to 1000 [ 0.320415] ACPI: Added _OSI(Module Device) [ 0.322027] ACPI: Added _OSI(Processor Device) [ 0.324022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.326023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.331285] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.340292] ACPI: Interpreter enabled [ 0.341062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.343014] ACPI: Using IOAPIC for interrupt routing [ 0.345126] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.350461] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.360539] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.363045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.366020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.369085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.373297] acpiphp: Slot [2] registered [ 0.376181] acpiphp: Slot [5] registered [ 0.377124] acpiphp: Slot [6] registered [ 0.379121] acpiphp: Slot [3] registered [ 0.381159] acpiphp: Slot [4] registered [ 0.382180] acpiphp: Slot [7] registered [ 0.384151] acpiphp: Slot [8] registered [ 0.386152] acpiphp: Slot [9] registered [ 0.388148] acpiphp: Slot [10] registered [ 0.389117] acpiphp: Slot [11] registered [ 0.391240] acpiphp: Slot [12] registered [ 0.393139] acpiphp: Slot [13] registered [ 0.394121] acpiphp: Slot [14] registered [ 0.396155] acpiphp: Slot [15] registered [ 0.398139] acpiphp: Slot [16] registered [ 0.400156] acpiphp: Slot [17] registered [ 0.401134] acpiphp: Slot [18] registered [ 0.403142] acpiphp: Slot [19] registered [ 0.404123] acpiphp: Slot [20] registered [ 0.406150] acpiphp: Slot [21] registered [ 0.407137] acpiphp: Slot [22] registered [ 0.409142] acpiphp: Slot [23] registered [ 0.411161] acpiphp: Slot [24] registered [ 0.412101] acpiphp: Slot [25] registered [ 0.414165] acpiphp: Slot [26] registered [ 0.415146] acpiphp: Slot [27] registered [ 0.417195] acpiphp: Slot [28] registered [ 0.419110] acpiphp: Slot [29] registered [ 0.420117] acpiphp: Slot [30] registered [ 0.422127] acpiphp: Slot [31] registered [ 0.424063] PCI host bridge to bus 0000:00 [ 0.425020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.427025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.430023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.432021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.435029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.438037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.440240] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.445207] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.448424] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.457623] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.463100] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.466029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.469034] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.472031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.474579] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.477790] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.480043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.483515] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.487781] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.498017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.502025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.508541] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.518038] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.526024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.556021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.566990] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.571014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.574989] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.585015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.593533] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.596356] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.598336] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.600316] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.602200] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.608037] iommu: Default domain type: Passthrough [ 0.610611] SCSI subsystem initialized [ 0.612202] ACPI: bus type USB registered [ 0.614181] usbcore: registered new interface driver usbfs [ 0.616160] usbcore: registered new interface driver hub [ 0.618126] usbcore: registered new device driver usb [ 0.620153] pps_core: LinuxPPS API ver. 1 registered [ 0.622015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.625063] PTP clock support registered [ 0.629043] EDAC MC: Ver: 3.0.0 [ 0.631192] PCI: Using ACPI for IRQ routing [ 0.633705] NetLabel: Initializing [ 0.635013] NetLabel: domain hash size = 128 [ 0.636009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.637079] NetLabel: unlabeled traffic allowed by default [ 0.639112] vgaarb: loaded [ 0.640354] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.642024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.648995] clocksource: Switched to clocksource kvm-clock [ 0.757265] VFS: Disk quotas dquot_6.6.0 [ 0.760573] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.763361] *** VALIDATE ramfs *** [ 0.764645] *** VALIDATE hugetlbfs *** [ 0.767154] pnp: PnP ACPI init [ 0.769841] pnp: PnP ACPI: found 6 devices [ 0.789811] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.793370] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.796041] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.798831] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.801637] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.804869] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.808255] NET: Registered protocol family 2 [ 0.811111] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.816477] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.821526] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.827817] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.831579] TCP: Hash tables configured (established 65536 bind 65536) [ 0.835083] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.838602] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.841982] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.845499] NET: Registered protocol family 1 [ 0.848905] RPC: Registered named UNIX socket transport module. [ 0.851364] RPC: Registered udp transport module. [ 0.853010] RPC: Registered tcp transport module. [ 0.854590] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.856573] NET: Registered protocol family 44 [ 0.857703] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.859280] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.861594] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.863769] PCI: CLS 0 bytes, default 64 [ 0.865448] Unpacking initramfs... [ 2.322694] debug: unmapping init [mem 0xffff966a3cc64000-0xffff966a3ffcffff] [ 2.326060] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.328860] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.331955] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.848362] Initialise system trusted keyrings [ 2.850442] Key type blacklist registered [ 2.852627] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.862211] zbud: loaded [ 2.865542] *** VALIDATE nfs *** [ 2.866886] *** VALIDATE nfs4 *** [ 2.868986] pstore: using deflate compression [ 2.872985] Platform Keyring initialized [ 2.978414] NET: Registered protocol family 38 [ 2.980421] Key type asymmetric registered [ 2.982075] Asymmetric key parser 'x509' registered [ 2.983675] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.986502] io scheduler mq-deadline registered [ 2.988265] io scheduler kyber registered [ 2.989739] io scheduler bfq registered [ 2.992243] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.995483] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.998436] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.001750] ACPI: Power Button [PWRF] [ 3.007437] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.013881] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.026217] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.053358] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.082076] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.087228] Non-volatile memory driver v1.3 [ 3.088681] Linux agpgart interface v0.103 [ 3.122309] virtio_blk virtio1: [vda] 149448 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.125241] vda: detected capacity change from 0 to 76517376 [ 3.138493] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.141148] vdb: detected capacity change from 0 to 1073741824 [ 3.147314] libphy: Fixed MDIO Bus: probed [ 3.155966] usbcore: registered new interface driver usbserial_generic [ 3.158946] usbserial: USB Serial support registered for generic [ 3.161495] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.166044] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.168089] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.170993] mousedev: PS/2 mouse device common for all mice [ 3.173382] rtc_cmos 00:05: RTC can wake from S4 [ 3.176201] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.178923] rtc_cmos 00:05: registered as rtc0 [ 3.181283] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.183419] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.185180] intel_pstate: CPU model not supported [ 3.191029] hid: raw HID events driver (C) Jiri Kosina [ 3.191489] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.193291] usbcore: registered new interface driver usbhid [ 3.197380] usbhid: USB HID core driver [ 3.198880] drop_monitor: Initializing network drop monitor service [ 3.201380] Initializing XFRM netlink socket [ 3.203394] NET: Registered protocol family 10 [ 3.206422] Segment Routing with IPv6 [ 3.207858] NET: Registered protocol family 17 [ 3.209578] mpls_gso: MPLS GSO support [ 3.214042] RAS: Correctable Errors collector initialized. [ 3.215802] AVX version of gcm_enc/dec engaged. [ 3.217398] AES CTR mode by8 optimization enabled [ 3.292828] sched_clock: Marking stable (3292803006, 0)->(4204792720, -911989714) [ 3.296635] registered taskstats version 1 [ 3.298136] Loading compiled-in X.509 certificates [ 3.300173] zswap: loaded using pool lzo/zbud [ 3.326056] Key type big_key registered [ 3.340279] Key type encrypted registered [ 3.342612] ima: No TPM chip found, activating TPM-bypass! [ 3.345111] ima: Allocated hash algorithm: sha1 [ 3.346881] ima: No architecture policies found [ 3.348537] evm: Initialising EVM extended attributes: [ 3.350629] evm: security.selinux [ 3.352163] evm: security.ima [ 3.353534] evm: security.capability [ 3.355206] evm: HMAC attrs: 0x1 [ 3.357693] rtc_cmos 00:05: setting system clock to 2026-08-22 07:53:29 UTC (1787385209) [ 3.364648] debug: unmapping init [mem 0xffffffffb0e03000-0xffffffffb0ffffff] [ 3.367080] debug: unmapping init [mem 0xffffffffafb82000-0xffffffffafe58fff] [ 3.378145] Write protecting the kernel read-only data: 28672k [ 3.380965] debug: unmapping init [mem 0xffffffffae203000-0xffffffffae3fffff] [ 3.384364] debug: unmapping init [mem 0xffffffffaeb14000-0xffffffffaebfffff] [ 3.418994] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.428318] systemd[1]: Detected virtualization kvm. [ 3.429641] systemd[1]: Detected architecture x86-64. [ 3.431459] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.457093] systemd[1]: No hostname configured. [ 3.458418] systemd[1]: Set hostname to . [ 3.460641] random: systemd: uninitialized urandom read (16 bytes read) [ 3.462896] systemd[1]: Initializing machine ID from random generator. [ 3.513522] random: ln: uninitialized urandom read (6 bytes read) [ 3.605132] random: systemd: uninitialized urandom read (16 bytes read) [ 3.608891] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.617049] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.622263] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.279794] device-mapper: uevent: version 1.0.3 [ 4.282456] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK [ 4.974973] random: fast init done ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.989523] virtio_net virtio0 ens2: renamed from eth0 [ 5.040279] scsi host0: ata_piix [ 5.060192] scsi host1: ata_piix [ 5.061879] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.064516] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.545699] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.860621] random: crng init done [ 9.862446] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.368063] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.583094] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.858855] SELinux: Disabled at runtime. [ 11.921478] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.931736] systemd[1]: Detected virtualization kvm. [ 11.935684] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.504168] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.508152] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.516655] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.521684] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.525440] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.540661] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.551429] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ 12.737742] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.049715] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.404160] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.489812] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.534596] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.556480] EDAC sbridge: Ver: 1.1.2 [ 14.821778] Key type dns_resolver registered [ 15.128802] NFS: Registering the id_resolver key type [ 15.130694] Key type id_resolver registered [ 15.132262] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg328-client login: [ 44.447797] libcfs: loading out-of-tree module taints kernel. [ 44.524227] Key type ._llcrypt registered [ 44.525900] Key type .llcrypt registered [ 44.955548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 44.963600] alg: No test for adler32 (adler32-zlib) [ 46.000042] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 46.378123] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 48.016207] Key type lgssc registered [ 48.772790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.830421] hrtimer: interrupt took 4270537 ns [ 185.769235] Lustre: Mounted lustre-client [ 191.212813] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 206.059522] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing check_logdir /tmp/testlogs/ [ 211.081362] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing yml_node [ 211.428867] Lustre: lustre-OST0000-osc-ffff966a98833000: disconnect after 23s idle [ 215.409787] Lustre: DEBUG MARKER: Client: 2.17.54.112 [ 218.537765] Lustre: DEBUG MARKER: MDS: 2.17.54.112 [ 221.526291] Lustre: DEBUG MARKER: OSS: 2.17.54.112 [ 223.341436] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Aug 22 03:57:07 EDT 2026 [ 241.651897] Lustre: Unmounted lustre-client [ 265.980188] Key type lgssc unregistered [ 266.246914] LNet: 8387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 266.263526] LNetError: 8387:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 266.275137] LNet: Removed LNI 192.168.203.28@tcp [ 267.144329] Key type .llcrypt unregistered [ 267.149728] Key type ._llcrypt unregistered [ 293.826144] Key type ._llcrypt registered [ 293.828896] Key type .llcrypt registered [ 293.874114] LNet: 8966:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 293.874114] mask usage: [+|-] ... [ 294.091740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 294.108939] alg: No test for adler32 (adler32-zlib) [ 295.023400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 295.095348] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 295.129207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 299.373690] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 299.497301] LNet: 9264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.503957] LNetError: 9264:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 299.530473] LNet: Removed LNI 192.168.203.28@tcp [ 300.657785] Key type .llcrypt unregistered [ 300.668507] Key type ._llcrypt unregistered [ 307.569585] Key type ._llcrypt registered [ 307.572226] Key type .llcrypt registered [ 308.999650] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 03:58:33 (1787385513) [ 309.264578] LNet: 9643:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 309.264578] mask usage: [+|-] ... [ 309.428760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 309.452577] alg: No test for adler32 (adler32-zlib) [ 310.432258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.608748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.652710] LNet: 9683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.683739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.789565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 310.824322] LNet: 9690:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 317.670852] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 03:58:42 (1787385522) [ 318.070318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.089444] LNet: 10267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.115695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.194960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 325.129495] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 03:58:49 (1787385529) [ 325.239406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.263959] LNet: 10848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.289644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.350429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 332.723497] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 03:58:57 (1787385537) [ 333.016950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.051295] LNet: 11438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.083357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.203231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 341.486681] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 03:59:06 (1787385546) [ 341.808728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 341.836329] LNet: 12023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.874846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.998891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 342.921475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 351.267392] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 03:59:15 (1787385555) [ 351.452198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.487536] LNet: 12611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.549634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.726944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 352.593532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 353.431420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 361.493527] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 03:59:26 (1787385566) [ 361.598419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.642530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.736498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 369.663985] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 03:59:34 (1787385574) [ 369.913510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.954726] LNet: 13797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.962575] LNet: 13797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 369.989945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.061204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 370.103617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 370.665993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 370.709985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 371.391056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 371.439165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 372.407646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 372.456197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 372.751889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 372.781971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 373.129102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 373.180731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 373.719823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 373.765515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 374.126294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 374.146963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 374.434968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 374.486128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 374.919127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 374.952367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 375.415462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 375.494324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 376.135131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 376.211942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 376.720573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 376.764434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 383.620831] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 03:59:48 (1787385588) [ 383.758701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.839199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.961906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 384.017947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 392.164199] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 03:59:56 (1787385596) [ 392.486725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.549057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.652599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 392.702355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 400.229907] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 04:00:04 (1787385604) [ 400.533866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.585982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.671931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 400.731295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 407.267823] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 04:00:12 (1787385612) [ 407.599349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.637807] LNet: 16239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.647267] LNet: 16239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 407.678873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.783485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 407.829899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 414.556737] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 04:00:19 (1787385619) [ 414.781414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.831132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.915369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 414.983905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 422.024802] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 04:00:26 (1787385626) [ 422.367749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.447432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.519487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 422.572639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 430.591858] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 04:00:34 (1787385634) [ 430.799590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.875806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.008502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 431.848344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 439.890589] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 04:00:44 (1787385644) [ 440.105828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.194286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.577392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 440.863050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 448.257339] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 04:00:52 (1787385652) [ 448.545758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.617752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.792817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 449.099705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 455.730038] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 04:01:00 (1787385660) [ 455.807410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.882606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.650923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 456.952377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 464.824174] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 04:01:09 (1787385669) [ 464.896721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.936098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.021280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 465.073520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 472.777930] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 04:01:17 (1787385677) [ 472.890505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.924311] LNet: 21084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 472.933199] LNet: 21084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 472.960106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.078493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 473.180959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 479.658483] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 04:01:24 (1787385684) [ 479.792929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.870472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.956944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 486.604324] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 04:01:31 (1787385691) [ 486.699209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.741046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.817529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 493.425406] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 04:01:38 (1787385698) [ 493.525536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.572965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.713231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 494.170734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 501.869570] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 04:01:46 (1787385706) [ 502.055498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.112132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.202661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 502.245112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 509.666821] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 04:01:54 (1787385714) [ 509.885949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.966841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.044139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 510.096325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 517.915673] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 04:02:02 (1787385722) [ 518.057657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.130649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.242993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 518.297846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 525.345179] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 04:02:10 (1787385730) [ 525.646062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.704063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.813883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 525.862877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 532.818446] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 04:02:17 (1787385737) [ 532.922793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.964863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.033214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 533.110988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 533.133743] LNetError: 25813:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 533.194616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 533.239389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 539.879616] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 04:02:24 (1787385744) [ 539.975906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.016203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.150248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.bnDjOfJjTG [ 540.228547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.bnDjOfJjTG [ 540.280273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.bnDjOfJjTG [ 546.752919] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 04:02:31 (1787385751) [ 547.070562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.128788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.204714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 547.263729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 547.329472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 553.333748] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 04:02:38 (1787385758) [ 553.774182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.827886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.894863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 556.414751] LNet: Message forwarding will be enabled in 2s [ 559.520456] LNet: Message forwarding enabled [ 559.872273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 559.893900] LNet: Message forwarding will stop in 2s to 6s [ 564.440133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 565.062991] LNet: Message forwarding disabled [ 566.873623] LNet: Message forwarding will be enabled in 2s [ 570.211385] LNet: Message forwarding enabled [ 573.733800] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 04:02:58 (1787385778) [ 573.872758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.351625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.440211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 581.214440] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 04:03:06 (1787385786) [ 581.418181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.452544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.498534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 581.524590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 581.555526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 581.580239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 581.616964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 581.639942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 581.661776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 581.687920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 581.714260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 581.736898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 581.776911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 581.820466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 581.852974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 581.895929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 581.933860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 581.971661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 582.006197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 582.039054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 582.071145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 582.093552] LNetError: 28885:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 588.280494] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 04:03:13 (1787385793) [ 588.638695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.674375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.767707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 595.852269] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 04:03:20 (1787385800) [ 598.131393] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg328-server [ 600.152625] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 04:03:24 (1787385804) [ 600.273600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.334225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.401488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 600.428135] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 602.508716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[29-33]@tcp [ 602.540535] LNet: 30602:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 602.620644] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.29@tcp is being used as a gateway but routing feature is not turned on [ 602.642172] LNetError: 30391:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.29@tcp has gone from up to down [ 603.620567] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.29@tcp is being used as a gateway but routing feature is not turned on [ 603.643533] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 604.647184] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.30@tcp is being used as a gateway but routing feature is not turned on [ 604.683157] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 606.696159] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.31@tcp is being used as a gateway but routing feature is not turned on [ 606.714273] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 609.460535] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 04:03:34 (1787385814) [ 610.793172] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.29@tcp is being used as a gateway but routing feature is not turned on [ 610.813910] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 611.728722] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg328-server [ 613.428427] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 04:03:38 (1787385818) [ 613.772705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.812615] LNet: 31537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 613.819675] LNet: 31537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 15 previous similar messages [ 613.830968] LNetError: 31537:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 613.857236] LNet: Removed LNI 192.168.203.28@tcp [ 613.887547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.938581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 613.957368] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 614.707667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[29-34/2]@tcp [ 614.725758] LNet: 31654:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 614.734377] LNet: 31654:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 614.747717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[29-34/2]@tcp [ 614.770235] LNetError: 31541:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.33@tcp has gone from up to down [ 614.781275] LNetError: 31541:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 622.127922] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 04:03:46 (1787385826) [ 622.242346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.267226] LNetError: 32241:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 622.301198] LNet: Removed LNI 192.168.203.28@tcp [ 622.381447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.488985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 622.591985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.641217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.715438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 622.779380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.816842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.859397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 622.924479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.955360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.012891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 623.125400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.175846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.238477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 623.321703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.358732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.447136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 623.558066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.616160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.685522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 623.799305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.859426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.979454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 624.082147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.148393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.204276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 624.316467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.368852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.456337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 624.537917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.600387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.638935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 624.699537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.735820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.792336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.806419] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 04:03:55 (1787385835) [ 632.821481] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg328-server [ 634.776041] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 04:03:59 (1787385839) [ 636.273435] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg328-server [ 638.563419] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 04:04:03 (1787385843) [ 639.399338] Key type .llcrypt unregistered [ 639.401585] Key type ._llcrypt unregistered [ 645.732542] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 645.915762] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 645.919383] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 645.922501] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 648.477114] Key type ._llcrypt registered [ 648.482204] Key type .llcrypt registered [ 648.518513] LNet: 33928:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 648.518513] mask usage: [+|-] ... [ 648.720481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 648.763215] alg: No test for adler32 (adler32-zlib) [ 649.692953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.753202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 649.790616] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 650.422716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 650.449832] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 652.485453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 652.512932] LNet: Removed LNI 10.1.2.3@tcp [ 652.658230] LNet: 34369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.667486] LNetError: 34369:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 652.687800] LNet: Removed LNI 192.168.203.28@tcp [ 653.448393] Key type .llcrypt unregistered [ 653.451256] Key type ._llcrypt unregistered [ 664.610195] Key type ._llcrypt registered [ 664.616811] Key type .llcrypt registered [ 665.874475] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 04:04:30 (1787385870) [ 666.393366] Key type .llcrypt unregistered [ 666.396175] Key type ._llcrypt unregistered [ 671.955735] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 672.020197] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 672.026941] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 672.031525] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 674.718613] Key type ._llcrypt registered [ 674.725036] Key type .llcrypt registered [ 674.802605] LNet: 35405:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 674.802605] mask usage: [+|-] ... [ 674.970225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 674.988972] alg: No test for adler32 (adler32-zlib) [ 675.951953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.020276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 676.036820] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 676.806535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 676.840815] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 679.279404] LNet: 35863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.303553] LNetError: 35863:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 679.311456] LNet: Removed LNI 10.1.2.3@tcp [ 679.339289] LNet: Removed LNI 192.168.203.28@tcp [ 679.946677] Key type .llcrypt unregistered [ 679.959200] Key type ._llcrypt unregistered [ 689.595944] Key type ._llcrypt registered [ 689.599221] Key type .llcrypt registered [ 690.677631] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 04:04:55 (1787385895) [ 691.181239] Key type .llcrypt unregistered [ 691.183045] Key type ._llcrypt unregistered [ 697.120088] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 697.275318] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 697.281697] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 697.291613] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 699.967600] Key type ._llcrypt registered [ 699.975073] Key type .llcrypt registered [ 700.021587] LNet: 36897:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 700.021587] mask usage: [+|-] ... [ 700.170621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 700.183905] alg: No test for adler32 (adler32-zlib) [ 701.104725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.175652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 701.212690] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 702.834193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 702.861186] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 704.708584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 704.753259] LNet: Removed LNI 10.1.2.3@tcp [ 704.771204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 704.809760] LNet: Removed LNI 10.1.2.31@tcp [ 704.827461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 704.848631] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 705.386959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 705.413206] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 706.462434] LNet: 37780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.471319] LNetError: 37780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 706.479242] LNet: Removed LNI 10.1.2.31@tcp [ 706.503525] LNet: Removed LNI 10.1.2.3@tcp [ 707.173339] Key type .llcrypt unregistered [ 707.175898] Key type ._llcrypt unregistered [ 720.864875] Key type ._llcrypt registered [ 720.873291] Key type .llcrypt registered [ 722.099648] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 04:05:26 (1787385926) [ 722.509624] LNet: 38573:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 722.509624] mask usage: [+|-] ... [ 722.731355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 722.769611] alg: No test for adler32 (adler32-zlib) [ 723.779459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.896303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 723.920815] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 725.058719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.092935] LNet: 38734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.105741] LNetError: 38734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.142978] LNet: Removed LNI 192.168.203.28@tcp [ 725.190241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.293631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.28@tcp [ 725.329741] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 726.321565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.28@tcp [ 726.361300] LNet: Removed LNI 192.168.203.28@tcp [ 726.478122] LNet: 38869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.129665] Key type .llcrypt unregistered [ 727.134167] Key type ._llcrypt unregistered [ 739.451125] Key type ._llcrypt registered [ 739.454125] Key type .llcrypt registered [ 741.245648] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 04:05:45 (1787385945) [ 742.917209] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg328-server [ 744.678239] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 04:05:49 (1787385949) [ 745.292591] Key type .llcrypt unregistered [ 745.299178] Key type ._llcrypt unregistered [ 751.294324] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 751.475471] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 751.489724] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 751.499945] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 754.168236] Key type ._llcrypt registered [ 754.174090] Key type .llcrypt registered [ 754.225360] LNet: 40242:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 754.225360] mask usage: [+|-] ... [ 754.419236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 754.440932] alg: No test for adler32 (adler32-zlib) [ 755.422963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.477032] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 756.172855] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 758.358671] LNet: 40702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.388142] LNetError: 40702:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.396837] LNet: Removed LNI 192.168.203.28@tcp1 [ 758.412354] LNet: Removed LNI 10.1.2.3@tcp2 [ 759.253301] Key type .llcrypt unregistered [ 759.280980] Key type ._llcrypt unregistered [ 772.835644] Key type ._llcrypt registered [ 772.839447] Key type .llcrypt registered [ 774.417466] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 04:06:18 (1787385978) [ 774.755838] LNet: 41506:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 774.755838] mask usage: [+|-] ... [ 774.905843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 774.922380] alg: No test for adler32 (adler32-zlib) [ 775.845737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.909523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 775.945303] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 776.972451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 776.994296] LNet: 41671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.002528] LNetError: 41671:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.021799] LNet: Removed LNI 192.168.203.28@tcp [ 777.037151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.109859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 777.153176] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 784.175462] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 04:06:29 (1787385989) [ 784.285625] LNet: 42256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.294625] LNetError: 42256:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.320443] LNet: Removed LNI 192.168.203.28@tcp [ 785.027503] Key type .llcrypt unregistered [ 785.036665] Key type ._llcrypt unregistered [ 791.312215] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 791.506509] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 791.525777] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 791.531228] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 794.059349] Key type ._llcrypt registered [ 794.061586] Key type .llcrypt registered [ 794.090566] LNet: 42543:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 794.090566] mask usage: [+|-] ... [ 794.228322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 794.247212] alg: No test for adler32 (adler32-zlib) [ 795.158940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.230275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 795.267420] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 796.057384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 796.079945] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 797.950506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.003591] LNet: 43009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.019891] LNetError: 43009:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 798.035842] LNet: Removed LNI 10.1.2.3@tcp [ 798.072617] LNet: Removed LNI 192.168.203.28@tcp [ 798.089870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.182951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 798.224809] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 798.251196] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 798.955341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.978580] LNet: 43036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.984330] LNetError: 43036:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 798.992831] LNetError: 43036:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 799.001173] LNet: Removed LNI 10.1.2.3@tcp [ 799.017966] LNet: Removed LNI 192.168.203.28@tcp [ 799.044651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.107630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 799.131569] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 799.135669] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 799.649207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 799.699836] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 800.775780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.799410] LNet: 43199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.813399] LNetError: 43199:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 800.820057] LNetError: 43199:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 800.828982] LNet: Removed LNI 10.1.2.3@tcp [ 800.837324] LNet: Removed LNI 192.168.203.28@tcp [ 800.850389] LNet: Removed LNI 192.168.203.28@tcp2 [ 800.880160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.983645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 801.024764] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 801.030314] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 801.034287] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 808.640446] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 04:06:53 (1787386013) [ 808.849662] LNet: 43788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.861923] LNetError: 43788:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 808.868424] LNetError: 43788:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 808.883016] LNet: Removed LNI 10.1.2.3@tcp [ 808.886492] LNet: Removed LNI 192.168.203.28@tcp [ 808.899762] LNet: Removed LNI 192.168.203.28@tcp2 [ 809.726520] Key type .llcrypt unregistered [ 809.730738] Key type ._llcrypt unregistered [ 817.134163] Key type ._llcrypt registered [ 817.140491] Key type .llcrypt registered [ 817.190516] LNet: 44022:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 817.190516] mask usage: [+|-] ... [ 817.420905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 817.450614] alg: No test for adler32 (adler32-zlib) [ 818.403725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.471550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 818.497528] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 819.217746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.231959] LNet: 44191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.237749] LNetError: 44191:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.251904] LNet: Removed LNI 192.168.203.28@tcp [ 819.285537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.333043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 819.362168] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 819.879661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.895682] LNet: 44221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.905922] LNetError: 44221:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.938150] LNet: Removed LNI 192.168.203.28@tcp [ 819.969285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.046073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 820.083812] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 820.629440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.662138] LNet: Removed LNI 192.168.203.28@tcp [ 820.706433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.827723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 820.873575] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 821.507135] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 821.740718] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 821.754432] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 821.776106] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 824.195813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.256945] LNet: 44331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.273037] LNet: 44331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 824.280401] LNetError: 44331:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.292414] LNetError: 44331:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 824.323420] LNet: Removed LNI 192.168.203.28@tcp [ 824.344686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.394346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 824.413870] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 832.107605] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 04:07:16 (1787386036) [ 832.250049] LNet: 44922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.262712] LNetError: 44922:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.285243] LNet: Removed LNI 192.168.203.28@tcp [ 833.034422] Key type .llcrypt unregistered [ 833.037565] Key type ._llcrypt unregistered [ 840.896536] Key type ._llcrypt registered [ 840.898194] Key type .llcrypt registered [ 840.941172] LNet: 45155:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 840.941172] mask usage: [+|-] ... [ 841.141224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 841.173238] alg: No test for adler32 (adler32-zlib) [ 842.158472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.251837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 842.309994] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 843.308707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.323878] LNet: 45328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.330406] LNetError: 45328:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.344673] LNet: Removed LNI 192.168.203.28@tcp [ 843.375216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.453936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 843.483137] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 850.763620] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 04:07:35 (1787386055) [ 850.891655] LNet: 45913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.904054] LNetError: 45913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.943677] LNet: Removed LNI 192.168.203.28@tcp [ 851.685764] Key type .llcrypt unregistered [ 851.688167] Key type ._llcrypt unregistered [ 858.388258] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 858.564983] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 858.570573] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 858.576616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 861.190381] Key type ._llcrypt registered [ 861.194121] Key type .llcrypt registered [ 861.243536] LNet: 46195:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 861.243536] mask usage: [+|-] ... [ 861.423715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 861.454533] alg: No test for adler32 (adler32-zlib) [ 862.376892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.453391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 862.494129] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 863.313460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.347496] LNet: 46368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.355201] LNetError: 46368:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.368721] LNet: Removed LNI 192.168.203.28@tcp [ 863.391461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.446973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 863.469255] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 869.480760] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 04:07:54 (1787386074) [ 869.836832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.895872] LNet: 46966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.909490] LNetError: 46966:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.942407] LNet: Removed LNI 192.168.203.28@tcp [ 869.982064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.117866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 878.326739] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 04:08:02 (1787386082) [ 878.425796] LNet: 47541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.481403] Key type .llcrypt unregistered [ 879.486260] Key type ._llcrypt unregistered [ 887.270432] Key type ._llcrypt registered [ 887.279027] Key type .llcrypt registered [ 887.333701] LNet: 47774:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 887.333701] mask usage: [+|-] ... [ 887.534102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 887.552503] alg: No test for adler32 (adler32-zlib) [ 888.594805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.677341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 888.733906] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 889.737443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.753674] LNet: 47946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.759510] LNetError: 47946:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.772668] LNet: Removed LNI 192.168.203.28@tcp [ 889.788971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.836276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 889.888339] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 896.503482] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 04:08:21 (1787386101) [ 896.929169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.983096] LNet: 48539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.001122] LNetError: 48539:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.042270] LNet: Removed LNI 192.168.203.28@tcp [ 897.077064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.182467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 903.533542] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 04:08:28 (1787386108) [ 904.958030] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 907.296374] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 04:08:31 (1787386111) [ 908.967625] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 910.635166] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 04:08:35 (1787386115) [ 913.287159] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 915.313642] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 04:08:39 (1787386119) [ 917.147112] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 918.988848] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 04:08:43 (1787386123) [ 921.177109] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 923.496270] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 04:08:47 (1787386127) [ 923.667578] LNet: 50589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.560495] Key type .llcrypt unregistered [ 924.566396] Key type ._llcrypt unregistered [ 932.241268] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 932.418141] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 932.431353] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 932.436148] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 935.195915] Key type ._llcrypt registered [ 935.200086] Key type .llcrypt registered [ 935.290226] LNet: 50876:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 935.290226] mask usage: [+|-] ... [ 935.517382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 935.546949] alg: No test for adler32 (adler32-zlib) [ 936.459680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.540929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 936.586072] LNet: Added LNI 192.168.203.28@tcp [8/512/0/180] [ 936.599207] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 936.601237] LNet: 50914:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 936.775670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.833627] LNet: 50933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.841989] LNetError: 50933:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 936.850102] LNet: Removed LNI 10.1.2.3@tcp [ 936.854376] LNetError: 50933:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.899140] LNet: Removed LNI 192.168.203.28@tcp [ 936.932449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.997209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 937.029059] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 937.143978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 937.197251] LNet: 50957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.208595] LNetError: 50957:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.228101] LNet: Removed LNI 192.168.203.28@tcp [ 937.275331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.381321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 937.445944] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 937.455484] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 937.463720] LNet: 50965:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 937.559983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 937.592245] LNet: 50983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.597599] LNetError: 50983:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 937.604882] LNet: Removed LNI 10.1.2.3@tcp [ 937.607423] LNetError: 50983:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.634188] LNet: Removed LNI 192.168.203.28@tcp [ 937.682724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.744692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 937.773976] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 937.785548] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 937.790592] LNet: 50991:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 938.170162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 938.203931] LNet: 51018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.215664] LNetError: 51018:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 938.226302] LNet: Removed LNI 10.1.2.3@tcp [ 938.228886] LNetError: 51018:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.258318] LNet: Removed LNI 192.168.203.28@tcp [ 938.308307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.394346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 938.417567] LNet: Added LNI 192.168.203.28@tcp [8/256/0/182] [ 938.421784] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 938.513526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 938.531960] LNet: 51046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.545963] LNetError: 51046:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 938.552978] LNet: Removed LNI 10.1.2.3@tcp [ 938.555193] LNetError: 51046:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.562912] LNet: Removed LNI 192.168.203.28@tcp [ 938.580712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.636314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 938.678481] LNet: Added LNI 192.168.203.28@tcp [8/256/0/182] [ 938.687915] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 938.695406] LNet: 51054:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 938.798624] LNet: Removed LNI 10.1.2.3@tcp [ 938.818067] LNet: Removed LNI 192.168.203.28@tcp [ 939.680334] Key type .llcrypt unregistered [ 939.682942] Key type ._llcrypt unregistered [ 952.228621] Key type ._llcrypt registered [ 952.231548] Key type .llcrypt registered [ 953.480801] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 04:09:18 (1787386158) [ 953.916110] LNet: 51869:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 953.916110] mask usage: [+|-] ... [ 954.016633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 954.025475] alg: No test for adler32 (adler32-zlib) [ 954.979797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.040576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 955.080412] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 955.988766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.28 [ 956.031075] LNet: 52030:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 956.044264] LNet: Removed LNI 192.168.203.28@tcp [ 962.722408] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 04:09:27 (1787386167) [ 963.026663] LNet: 52600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.031990] LNetError: 52600:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.047914] LNet: Removed LNI 192.168.203.28@tcp [ 963.600604] Key type .llcrypt unregistered [ 963.604453] Key type ._llcrypt unregistered [ 970.412181] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 970.717934] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 973.633452] Key type ._llcrypt registered [ 973.636381] Key type .llcrypt registered [ 973.696214] LNet: 52883:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 973.696214] mask usage: [+|-] ... [ 973.926085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 973.961501] alg: No test for adler32 (adler32-zlib) [ 975.551551] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 982.366129] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 04:09:47 (1787386187) [ 982.852827] LNet: 53515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.859522] LNetError: 53515:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 982.886558] LNet: Removed LNI 10.1.2.3@tcp [ 983.548569] Key type .llcrypt unregistered [ 983.553978] Key type ._llcrypt unregistered [ 990.795589] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 991.009038] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 991.020841] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 991.032578] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 993.734985] Key type ._llcrypt registered [ 993.737597] Key type .llcrypt registered [ 993.766749] LNet: 53788:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 993.766749] mask usage: [+|-] ... [ 993.882102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 993.894735] alg: No test for adler32 (adler32-zlib) [ 994.884986] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1002.057751] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 04:10:06 (1787386206) [ 1003.722066] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1005.576242] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 04:10:10 (1787386210) [ 1005.905748] LNet: 54700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.911042] LNetError: 54700:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1005.933473] LNet: Removed LNI 10.1.2.3@tcp [ 1006.552311] Key type .llcrypt unregistered [ 1006.558607] Key type ._llcrypt unregistered [ 1012.542453] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1012.712899] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1012.716162] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1012.720208] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1015.395642] Key type ._llcrypt registered [ 1015.398619] Key type .llcrypt registered [ 1015.433516] LNet: 54974:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1015.433516] mask usage: [+|-] ... [ 1015.595506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1015.611559] alg: No test for adler32 (adler32-zlib) [ 1016.578806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1016.647571] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1024.159525] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 04:10:29 (1787386229) [ 1024.561604] LNet: 55929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.569869] LNetError: 55929:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1024.590890] LNet: Removed LNI 10.1.2.3@tcp [ 1025.133141] Key type .llcrypt unregistered [ 1025.135517] Key type ._llcrypt unregistered [ 1030.087363] Key type ._llcrypt registered [ 1030.089484] Key type .llcrypt registered [ 1030.123502] LNet: 56152:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1030.123502] mask usage: [+|-] ... [ 1030.248245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1030.272614] alg: No test for adler32 (adler32-zlib) [ 1031.183879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1031.218679] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1037.913525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1038.484971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_interrupt -r 1 [ 1038.510703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_interrupt -r 1 [ 1038.542669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1038.557641] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1038.748134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1038.776827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1038.817534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1038.863722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1038.905886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1038.940215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1038.973280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1039.017480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1039.038238] LNet: 56366:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1039.052271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1039.101650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1039.130970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1039.157730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1039.194902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1039.246546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1039.290839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1039.765489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_dropped -r 1 [ 1039.795866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_dropped -r 1 [ 1039.825623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1039.844247] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1040.019442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1040.046486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1040.069838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1040.094654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1040.125465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1040.148354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1040.174343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1040.199782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1040.221166] LNet: 56475:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1040.247276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1040.285981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1040.322521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1040.378712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1040.412964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1040.444186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1040.477319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1040.997531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_aborted -r 1 [ 1041.023401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_aborted -r 1 [ 1041.053089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1041.071630] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1041.295251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1041.326628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1041.364241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1041.392076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1041.462276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1041.504284] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1041.522554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1041.572812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1041.609547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1041.626344] LNet: 56585:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1041.641597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1041.669251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1041.693151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1041.714795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1041.736720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1041.772616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1041.811683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1042.296068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_no_route -r 1 [ 1042.350271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_no_route -r 1 [ 1042.410330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1042.659796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1042.685290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1042.711469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1042.738755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1042.762576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1042.785636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1042.809370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1042.834059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1042.859982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1042.889756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1042.912523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1042.938823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1042.962318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1042.988569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1043.013981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1043.519304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_timeout -r 1 [ 1043.575626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_timeout -r 1 [ 1043.606531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1043.622892] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1043.627538] LNet: Skipped 2 previous similar messages [ 1043.845689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1043.873785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1043.901872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1043.928773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1043.953943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1043.979457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1044.007205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1044.030854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1044.057891] LNet: 56803:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1044.068269] LNet: 56803:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1044.085538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1044.123631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1044.157904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1044.183649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1044.217870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1044.249935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1044.286590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1044.782878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_error -r 1 [ 1044.832118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_error -r 1 [ 1044.858659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1045.111881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1045.154620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1045.187994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1045.210938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1045.235171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1045.260257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1045.285572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1045.318542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1045.365577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1045.406233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1045.442792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1045.473385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1045.518363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1045.553444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1045.597568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1045.865558] LNet: 56953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.875580] LNetError: 56953:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1045.896440] LNet: Removed LNI 192.168.203.28@tcp [ 1046.305941] Key type .llcrypt unregistered [ 1046.315125] Key type ._llcrypt unregistered [ 1064.469699] Key type ._llcrypt registered [ 1064.471719] Key type .llcrypt registered [ 1066.422462] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 04:11:10 (1787386270) [ 1067.697390] Key type .llcrypt unregistered [ 1067.709489] Key type ._llcrypt unregistered [ 1074.455525] Key type ._llcrypt registered [ 1074.457220] Key type .llcrypt registered [ 1074.573994] LNet: 57990:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1074.573994] mask usage: [+|-] ... [ 1074.902858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1074.932550] alg: No test for adler32 (adler32-zlib) [ 1075.833590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1075.887129] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1084.442917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1095.092951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1095.144296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1095.204126] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1096.803888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_interrupt -r 1 [ 1096.859687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_interrupt -r 1 [ 1096.894832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_interrupt -r 1 [ 1096.934428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_interrupt -r 1 [ 1096.982443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_interrupt -r 1 [ 1097.034817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_interrupt -r 1 [ 1097.072943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_interrupt -r 1 [ 1097.121759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_interrupt -r 1 [ 1097.200440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1097.225481] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1097.240734] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1097.484594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1097.530531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1097.563958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1097.587823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1097.624810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1097.654621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1097.682424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1097.717559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1097.743716] LNet: 58420:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1097.771367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1097.811985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1097.851288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1097.896973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1097.931253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1097.963816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1098.004672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1098.474220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_dropped -r 1 [ 1098.523246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_dropped -r 1 [ 1098.551736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_dropped -r 1 [ 1098.578202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_dropped -r 1 [ 1098.609188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_dropped -r 1 [ 1098.647836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_dropped -r 1 [ 1098.674792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_dropped -r 1 [ 1098.714140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_dropped -r 1 [ 1098.742921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1098.760928] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1098.765736] LNet: Skipped 2 previous similar messages [ 1098.961376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1098.990469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1099.023341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1099.064019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1099.092626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1099.113214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1099.135586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1099.159531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1099.190282] LNet: 58541:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1099.207785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1099.235331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1099.256800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1099.277058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1099.309767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1099.347770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1099.384877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1099.948380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_aborted -r 1 [ 1099.987183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_aborted -r 1 [ 1100.030921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_aborted -r 1 [ 1100.062309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_aborted -r 1 [ 1100.104355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_aborted -r 1 [ 1100.147917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_aborted -r 1 [ 1100.218104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_aborted -r 1 [ 1100.263926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_aborted -r 1 [ 1100.307183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1100.338760] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1100.346382] LNet: Skipped 2 previous similar messages [ 1100.542928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1100.574412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1100.622811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1100.656594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.690972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.726951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.759878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.807894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.832703] LNet: 58662:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.858553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.895794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.944214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1100.975985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1101.021517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1101.062100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1101.091334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1101.583888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_no_route -r 1 [ 1101.617265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_no_route -r 1 [ 1101.650354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_no_route -r 1 [ 1101.692057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_no_route -r 1 [ 1101.732496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_no_route -r 1 [ 1101.766737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_no_route -r 1 [ 1101.804036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_no_route -r 1 [ 1101.833376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_no_route -r 1 [ 1101.874501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1102.130599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1102.163906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1102.215186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1102.256589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.308743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1102.356449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1102.388491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1102.430166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1102.467274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1102.502220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1102.543439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1102.597698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1102.643874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1102.696775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1102.755402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1103.393837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_timeout -r 1 [ 1103.466866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_timeout -r 1 [ 1103.496950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_timeout -r 1 [ 1103.530497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_timeout -r 1 [ 1103.558561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_timeout -r 1 [ 1103.585065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_timeout -r 1 [ 1103.611537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_timeout -r 1 [ 1103.643887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_timeout -r 1 [ 1103.689819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1103.705973] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1103.710457] LNet: Skipped 5 previous similar messages [ 1103.943871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1103.974496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1104.013680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.042829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.070740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.111321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.159840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.196447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.233947] LNet: 58905:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.239074] LNet: 58905:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1104.266421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.300256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.346140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.371339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.397782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.426492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.463088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1105.049853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_error -r 1 [ 1105.094874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_error -r 1 [ 1105.147528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_error -r 1 [ 1105.189267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_error -r 1 [ 1105.239100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_error -r 1 [ 1105.306238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_error -r 1 [ 1105.346459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_error -r 1 [ 1105.405867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_error -r 1 [ 1105.460434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1105.688969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.721813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1105.764788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1105.829785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1105.875892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1105.922232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1105.953485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.988891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.029364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.063613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.102442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.139691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.176215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.242673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.304058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1112.225659] LNet: 59102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.232755] LNetError: 59102:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1112.240296] LNet: Removed LNI 192.168.203.28@tcp [ 1112.251154] LNet: Removed LNI 192.168.203.28@tcp1 [ 1112.672794] Key type .llcrypt unregistered [ 1112.678630] Key type ._llcrypt unregistered [ 1127.555589] Key type ._llcrypt registered [ 1127.558953] Key type .llcrypt registered [ 1128.785191] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 04:12:13 (1787386333) [ 1129.434600] Key type .llcrypt unregistered [ 1129.439330] Key type ._llcrypt unregistered [ 1135.443336] Key type ._llcrypt registered [ 1135.447640] Key type .llcrypt registered [ 1135.481920] LNet: 60146:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1135.481920] mask usage: [+|-] ... [ 1135.657791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1135.688239] alg: No test for adler32 (adler32-zlib) [ 1136.639730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1136.696676] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1143.548936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1144.134485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e remote_dropped -r 1 [ 1144.176414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e remote_dropped -r 1 [ 1144.214242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1144.229556] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1144.556296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1144.595259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1144.631675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1144.665421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1144.703180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1144.742973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1144.779246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1144.812365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1144.837723] LNet: 60361:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1144.863956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1144.898076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1144.924207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1144.949817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1144.971403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1145.007913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1145.037083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1145.522845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e remote_error -r 1 [ 1145.557438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e remote_error -r 1 [ 1145.590497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1145.621590] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1145.824678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1145.868687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1145.910414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1145.949522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1145.987308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1146.051756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1146.077075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1146.099479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1146.115499] LNet: 60470:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1146.140616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1146.183239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1146.222725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1146.253438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1146.273347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1146.294317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1146.332433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1146.782480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e remote_timeout -r 1 [ 1146.816296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e remote_timeout -r 1 [ 1146.875033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1146.901632] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1147.172736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1147.220949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1147.276959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1147.306822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1147.381070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1147.416602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1147.489385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1147.525909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1147.544780] LNet: 60579:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1147.565325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1147.585073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1147.616478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1147.650312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1147.683636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1147.710561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1147.739550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1148.085831] LNet: 60620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.091413] LNetError: 60620:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.121948] LNet: Removed LNI 192.168.203.28@tcp [ 1148.484415] Key type .llcrypt unregistered [ 1148.487757] Key type ._llcrypt unregistered [ 1165.956550] Key type ._llcrypt registered [ 1165.961897] Key type .llcrypt registered [ 1167.088261] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 04:12:51 (1787386371) [ 1167.939391] Key type .llcrypt unregistered [ 1167.943728] Key type ._llcrypt unregistered [ 1174.094744] Key type ._llcrypt registered [ 1174.097544] Key type .llcrypt registered [ 1174.131894] LNet: 61659:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1174.131894] mask usage: [+|-] ... [ 1174.285717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1174.295816] alg: No test for adler32 (adler32-zlib) [ 1175.242565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1175.266935] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1183.133174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1195.151882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1195.208428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1195.239389] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1196.492347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e remote_dropped -r 1 [ 1196.517675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e remote_dropped -r 1 [ 1196.553839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e remote_dropped -r 1 [ 1196.600296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e remote_dropped -r 1 [ 1196.643520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e remote_dropped -r 1 [ 1196.673639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e remote_dropped -r 1 [ 1196.702429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e remote_dropped -r 1 [ 1196.736046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e remote_dropped -r 1 [ 1196.767468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1196.786110] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1196.789233] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.128@tcp [ 1196.957667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1196.988272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1197.016952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1197.066465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1197.115324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1197.178822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1197.235700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1197.298898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1197.320590] LNet: 62089:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1197.339779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1197.370168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1197.401445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1197.439570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1197.487682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1197.543430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1197.588859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1197.632943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1198.188846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e remote_error -r 1 [ 1198.228376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e remote_error -r 1 [ 1198.276324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e remote_error -r 1 [ 1198.322349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e remote_error -r 1 [ 1198.372920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e remote_error -r 1 [ 1198.459457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e remote_error -r 1 [ 1198.518319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e remote_error -r 1 [ 1198.610883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e remote_error -r 1 [ 1198.678380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1198.707424] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1198.718378] LNet: Skipped 2 previous similar messages [ 1199.053113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1199.080287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1199.123348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1199.154757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1199.192501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1199.222817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1199.289109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1199.356552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1199.385727] LNet: 62213:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1199.405879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1199.447666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1199.474812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1199.499595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1199.526207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1199.549430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1199.582880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1199.630956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1200.193029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e remote_timeout -r 1 [ 1200.263073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e remote_timeout -r 1 [ 1200.333440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e remote_timeout -r 1 [ 1200.391312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e remote_timeout -r 1 [ 1200.454379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e remote_timeout -r 1 [ 1200.507688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e remote_timeout -r 1 [ 1200.597092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e remote_timeout -r 1 [ 1200.655190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e remote_timeout -r 1 [ 1200.686701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1200.716155] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1200.937861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1200.963850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1200.988286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1201.012173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1201.042674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1201.069117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1201.115643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1201.169158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1201.198312] LNet: 62336:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1201.219587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1201.276783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1201.329528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1201.356351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1201.408507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1201.437627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1201.488580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1201.545287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1206.578103] LNet: 62414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1206.588913] LNetError: 62414:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1206.596335] LNet: Removed LNI 192.168.203.28@tcp [ 1206.608802] LNet: Removed LNI 192.168.203.28@tcp1 [ 1207.183624] Key type .llcrypt unregistered [ 1207.190756] Key type ._llcrypt unregistered [ 1226.535881] Key type ._llcrypt registered [ 1226.540291] Key type .llcrypt registered [ 1228.205213] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 04:13:52 (1787386432) [ 1229.955235] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1231.787779] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 04:13:56 (1787386436) [ 1232.604865] Key type .llcrypt unregistered [ 1232.622684] Key type ._llcrypt unregistered [ 1238.619638] Key type ._llcrypt registered [ 1238.626267] Key type .llcrypt registered [ 1238.667820] LNet: 63745:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1238.667820] mask usage: [+|-] ... [ 1238.834834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1238.856850] alg: No test for adler32 (adler32-zlib) [ 1239.769831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1239.791332] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1247.234119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1248.184840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e network_timeout -r 1 [ 1248.243622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e network_timeout -r 1 [ 1248.276862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1248.288447] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1248.430755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1248.468910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1248.508149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1248.543077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1248.576416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1248.606434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1248.661923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1248.700641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1248.720055] LNet: 63961:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1248.737443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1248.771682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1248.804951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1248.846039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1248.900110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.963956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1248.993139] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1249.218278] LNet: 64000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1249.223301] LNetError: 64000:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.241948] LNet: Removed LNI 192.168.203.28@tcp [ 1249.560616] Key type .llcrypt unregistered [ 1249.563268] Key type ._llcrypt unregistered [ 1268.501691] Key type ._llcrypt registered [ 1268.503605] Key type .llcrypt registered [ 1268.533489] LNet: 64451:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1268.533489] mask usage: [+|-] ... [ 1268.679357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1268.692703] alg: No test for adler32 (adler32-zlib) [ 1269.600717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1269.619955] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1277.379879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1286.640714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1286.696549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1286.742643] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1287.944087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e network_timeout -r 1 [ 1287.984551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e network_timeout -r 1 [ 1288.037493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e network_timeout -r 1 [ 1288.097141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e network_timeout -r 1 [ 1288.156379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e network_timeout -r 1 [ 1288.199166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e network_timeout -r 1 [ 1288.236774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e network_timeout -r 1 [ 1288.281406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e network_timeout -r 1 [ 1288.339807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1288.358282] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1288.605878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1288.643614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1288.689959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1288.719967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1288.758975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1288.798846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1288.854530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1288.889665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1288.908950] LNet: 64881:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1288.929434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1288.970211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1289.008083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1289.044914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1289.056892] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1289.058586] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1289.065332] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.128@tcp [ 1289.090638] LNet: Skipped 1 previous similar message [ 1289.095882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1289.186275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1293.883303] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1294.406958] LNet: 64955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1294.415938] LNetError: 64955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1294.424867] LNet: Removed LNI 192.168.203.28@tcp [ 1294.437630] LNet: Removed LNI 192.168.203.28@tcp1 [ 1294.887483] Key type .llcrypt unregistered [ 1294.890626] Key type ._llcrypt unregistered [ 1309.639713] Key type ._llcrypt registered [ 1309.648355] Key type .llcrypt registered [ 1310.965968] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 04:15:15 (1787386515) [ 1311.230456] LNet: 65755:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1311.230456] mask usage: [+|-] ... [ 1311.433475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1311.447157] alg: No test for adler32 (adler32-zlib) [ 1312.411803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1312.481474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1312.521416] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1313.434696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1313.481175] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1314.572816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 1314.651877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1314.696774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1314.746645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1314.798619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1314.862302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1314.933950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.28@tcp [ 1314.966524] LNetError: 66057:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.203.28@tcp to recovery [ 1315.488619] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1315.489568] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1317.536331] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1318.494907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1321.632546] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1322.020574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1322.055718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1322.110283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1322.127971] LNet: 66165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1322.133693] LNetError: 66165:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1323.168124] LNet: Removed LNI 192.168.203.28@tcp [ 1323.170675] LNetError: 66165:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1323.184906] LNet: Removed LNI 192.168.203.28@tcp1 [ 1323.221808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1323.277051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1323.295426] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1323.909503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1323.938449] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1324.623088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 1324.664104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1324.734971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1324.773307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1324.802968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1324.837455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1324.865776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1324.910692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.28@tcp [ 1324.942648] LNetError: 66438:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.203.28@tcp to recovery [ 1325.280678] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1328.450987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1331.425106] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1331.437288] LNet: Skipped 1 previous similar message [ 1336.408848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1336.442962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1336.474889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1343.135592] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 04:15:47 (1787386547) [ 1343.260811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1343.293830] LNet: 67149:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1343.301516] LNetError: 67149:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.311066] LNet: Removed LNI 192.168.203.28@tcp [ 1343.328153] LNet: Removed LNI 192.168.203.28@tcp1 [ 1343.373320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1343.418427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1343.453472] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1344.129882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1344.164772] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1344.949556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 1345.017790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1345.054923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1345.099516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1345.133404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1345.166580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1345.205855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.28@tcp --health 0 [ 1345.306866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1345.440177] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.28@tcp [ 1346.468496] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1346.477890] LNet: Skipped 1 previous similar message [ 1361.633963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1361.783522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1361.814977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1361.854337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.28@tcp --health 500 [ 1361.975539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1368.600027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1368.702524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1368.736438] LNet: 67654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1368.753252] LNetError: 67654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1368.758338] LNetError: 67654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1368.765720] LNet: Removed LNI 192.168.203.28@tcp [ 1368.778985] LNet: Removed LNI 192.168.203.28@tcp1 [ 1368.806442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1368.871974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1368.892692] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1369.633833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1369.660562] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1370.587468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 1370.617887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1370.664650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1370.711154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1370.759996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1370.821301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1370.876549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1370.905062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.28@tcp --health 0 [ 1371.936351] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1371.950486] LNet: Skipped 3 previous similar messages [ 1372.257623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1381.499589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1381.544253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1381.606113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1388.437448] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 04:16:33 (1787386593) [ 1389.086523] LNet: 68688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.094368] LNetError: 68688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.102700] LNetError: 68688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1389.109134] LNet: Removed LNI 192.168.203.28@tcp [ 1389.123910] LNet: Removed LNI 192.168.203.28@tcp1 [ 1389.789664] Key type .llcrypt unregistered [ 1389.794845] Key type ._llcrypt unregistered [ 1396.049773] Key type ._llcrypt registered [ 1396.053682] Key type .llcrypt registered [ 1396.102659] LNet: 68911:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1396.102659] mask usage: [+|-] ... [ 1396.288642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1396.307420] alg: No test for adler32 (adler32-zlib) [ 1397.319727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1397.349600] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1407.058318] Lustre: DEBUG MARKER: Initial discovery [ 1407.244068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.128@tcp [ 1409.909136] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1410.262396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1410.314196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.128@tcp [ 1410.338862] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1410.370227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1410.416696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.128@tcp -r 1 -m GET -l 3 [ 1410.471877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.128@tcp -r 1 -m GET -e local_error [ 1410.520526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.128@tcp -r 1 -m PUT -l 6 [ 1410.661068] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1411.904613] Lustre: DEBUG MARKER: Force oleg328-client.virtnet to discover 192.168.203.128@tcp (in background) [ 1412.144706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.128@tcp [ 1413.668181] Lustre: DEBUG MARKER: Force oleg328-server to discover 192.168.203.28@tcp [ 1415.205680] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1416.423162] Lustre: DEBUG MARKER: Wait for 69451 [ 1417.759362] Lustre: DEBUG MARKER: Finished wait on 69451 [ 1418.174299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1418.229350] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1418.234462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1418.479246] LNet: 69900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1418.486226] LNetError: 69900:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1419.564843] LNet: Removed LNI 192.168.203.28@tcp [ 1419.994298] Key type .llcrypt unregistered [ 1419.997545] Key type ._llcrypt unregistered [ 1436.922619] Key type ._llcrypt registered [ 1436.925154] Key type .llcrypt registered [ 1438.344323] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 04:17:22 (1787386642) [ 1438.928710] Key type .llcrypt unregistered [ 1438.930803] Key type ._llcrypt unregistered [ 1446.508444] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1446.651227] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1446.658117] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1446.662733] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1449.260062] Key type ._llcrypt registered [ 1449.265743] Key type .llcrypt registered [ 1449.316775] LNet: 70944:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1449.316775] mask usage: [+|-] ... [ 1449.492411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1449.509741] alg: No test for adler32 (adler32-zlib) [ 1450.482979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.548343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1450.604410] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1451.384397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1451.405396] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1460.043254] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 04:17:44 (1787386664) [ 1460.325454] LNet: 71977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1460.334381] LNetError: 71977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1460.344470] LNet: Removed LNI 10.1.2.3@tcp [ 1460.388745] LNet: Removed LNI 192.168.203.28@tcp [ 1460.927475] Key type .llcrypt unregistered [ 1460.930407] Key type ._llcrypt unregistered [ 1467.217551] Key type ._llcrypt registered [ 1467.220591] Key type .llcrypt registered [ 1467.251854] LNet: 72227:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1467.251854] mask usage: [+|-] ... [ 1467.372859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1467.394805] alg: No test for adler32 (adler32-zlib) [ 1468.350447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1468.402591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1468.434511] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1469.287546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1469.305599] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1470.368604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.28@tcp [ 1471.456248] LNetError: Unexpected error -22 connecting to 192.168.203.28@tcp at host 192.168.203.28:988 [ 1476.679513] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 04:18:01 (1787386681) [ 1477.147697] LNet: 73110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1477.167456] LNetError: 73110:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1477.186965] LNet: Removed LNI 10.1.2.3@tcp [ 1477.225623] LNet: Removed LNI 192.168.203.28@tcp [ 1478.283478] Key type .llcrypt unregistered [ 1478.286469] Key type ._llcrypt unregistered [ 1485.429147] Key type ._llcrypt registered [ 1485.439274] Key type .llcrypt registered [ 1485.517780] LNet: 73342:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1485.517780] mask usage: [+|-] ... [ 1485.814078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1485.836319] alg: No test for adler32 (adler32-zlib) [ 1486.798019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.879943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1486.906076] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1487.559740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1487.578747] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 1488.665057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.28@tcp1 --nid 192.168.203.28@tcp2 [ 1492.882894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp1 192.168.203.28@tcp2 [ 1492.913606] LNetError: 73946:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.28@tcp1 to 192.168.203.28@tcp2 - Route table may be misconfigured [ 1492.936582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp2 192.168.203.28@tcp1 [ 1500.080286] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 04:18:24 (1787386704) [ 1500.412289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1500.444605] LNet: 74519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.453423] LNetError: 74519:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.467385] LNet: Removed LNI 192.168.203.28@tcp1 [ 1501.485451] LNet: Removed LNI 192.168.203.28@tcp2 [ 1501.522303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.614300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1501.664715] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1502.833587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1502.907404] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1503.773581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 1503.803940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp 192.168.203.28@tcp [ 1503.825981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp1 192.168.203.28@tcp1 [ 1503.864806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.28@tcp -d 192.168.203.28@tcp -e network_timeout [ 1503.887952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -e network_timeout [ 1503.917665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -e network_timeout [ 1503.941933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -e network_timeout [ 1503.973831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1503.987444] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1504.004835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1504.087508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1504.160437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1504.608751] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1510.801287] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 04:18:35 (1787386715) [ 1511.206751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1511.236640] LNet: 75381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.244356] LNetError: 75381:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.250687] LNetError: 75381:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1511.256417] LNet: Removed LNI 192.168.203.28@tcp [ 1512.296156] LNet: Removed LNI 192.168.203.28@tcp1 [ 1512.386609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1512.521396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1512.823924] LNet: 75418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.203485] Key type .llcrypt unregistered [ 1513.208084] Key type ._llcrypt unregistered [ 1525.354957] Key type ._llcrypt registered [ 1525.359199] Key type .llcrypt registered [ 1526.690889] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 04:18:51 (1787386731) [ 1527.019947] LNet: 76189:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1527.019947] mask usage: [+|-] ... [ 1527.173540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1527.187642] alg: No test for adler32 (adler32-zlib) [ 1528.074861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.571321] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1531.400639] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 04:18:56 (1787386736) [ 1531.484328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1531.513914] LNet: 76511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1531.536689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1531.603904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1531.648293] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1532.444099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1532.460525] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1533.348884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1533.392427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp1 [ 1533.434549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp1 [ 1539.382861] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 04:19:04 (1787386744) [ 1540.904244] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg328-server [ 1542.540326] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 04:19:07 (1787386747) [ 1544.289783] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg328-server [ 1545.887640] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 04:19:10 (1787386750) [ 1547.727281] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg328-server [ 1549.310742] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 04:19:14 (1787386754) [ 1551.454851] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg328-server [ 1553.161285] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 04:19:18 (1787386758) [ 1555.082945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg328-server [ 1556.945815] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 04:19:21 (1787386761) [ 1558.873712] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg328-server [ 1561.329597] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 04:19:25 (1787386765) [ 1563.617890] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg328-server [ 1565.740813] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 04:19:30 (1787386770) [ 1567.737104] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg328-server [ 1569.629992] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 04:19:34 (1787386774) [ 1571.953193] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg328-server [ 1573.871122] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 04:19:38 (1787386778) [ 1574.213933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.248727] LNet: 80396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1574.254095] LNetError: 80396:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.264191] LNet: Removed LNI 192.168.203.28@tcp [ 1575.266211] LNetError: 80396:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.292415] LNet: Removed LNI 192.168.203.28@tcp1 [ 1575.338464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.418987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1575.454740] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1576.295958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1576.440212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1576.732062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.757420] LNet: 80544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.764903] LNetError: 80544:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1577.830765] LNet: Removed LNI 192.168.203.28@tcp [ 1577.868655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.968482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1577.994540] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1579.123376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1579.384093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.423127] LNet: 80689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.434030] LNetError: 80689:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1580.468184] LNet: Removed LNI 192.168.203.28@tcp [ 1580.516642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.604697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1580.641397] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1581.586797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1581.700251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1581.970805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1583.078943] LNet: Removed LNI 192.168.203.28@tcp [ 1583.171199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.326199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1583.377888] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1584.367482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1584.596477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1584.616042] LNet: 80982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1584.623156] LNet: 80982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1584.629927] LNetError: 80982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1584.636653] LNetError: 80982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1584.673350] LNet: Removed LNI 192.168.203.28@tcp [ 1584.732260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.818933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1584.862734] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1585.907117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1586.001858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1586.323859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.369346] LNet: Removed LNI 192.168.203.28@tcp [ 1587.410497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.489374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1587.549549] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1588.558844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1590.117878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.206034] LNet: Removed LNI 192.168.203.28@tcp [ 1591.258951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.375951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1591.445952] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1592.580151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1592.794331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1594.322548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.370673] LNet: 81440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1594.378438] LNet: 81440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1594.401882] LNetError: 81440:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1594.408476] LNetError: 81440:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1594.433636] LNet: Removed LNI 192.168.203.28@tcp [ 1594.455680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.491990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1594.509806] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1595.782477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1597.324086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.378092] LNet: Removed LNI 192.168.203.28@tcp [ 1598.411912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.537408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1598.614569] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1599.637449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1599.746044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1601.222949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.293768] LNet: Removed LNI 192.168.203.28@tcp [ 1602.366609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1602.478185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1602.546679] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1603.796186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1604.167956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.222758] LNet: Removed LNI 192.168.203.28@tcp [ 1605.253966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.345991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1605.363909] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1606.286675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1606.343837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1606.665308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.722631] LNet: Removed LNI 192.168.203.28@tcp [ 1607.742737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.879943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1607.957113] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1609.253924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1610.648223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.664981] LNet: 82198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.669359] LNet: 82198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1610.688546] LNetError: 82198:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.703673] LNetError: 82198:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1611.753971] LNet: Removed LNI 192.168.203.28@tcp [ 1611.786517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.869911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1611.898701] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1612.918191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1613.015389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.071262] LNet: Removed LNI 192.168.203.28@tcp [ 1613.099286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.205415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1613.235462] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1614.303907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1621.608522] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 04:20:26 (1787386826) [ 1622.326248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.347671] LNet: Removed LNI 192.168.203.28@tcp [ 1622.357842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.402715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1622.428444] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/180] [ 1623.597882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.635869] LNet: Removed LNI 192.168.203.28@tcp231 [ 1623.653448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.798515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1623.877612] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/180] [ 1623.880766] LNet: 83181:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1624.370995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1624.440241] LNet: Removed LNI 192.168.203.28@tcp231 [ 1624.470246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1624.560820] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/180] [ 1626.035571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1626.064950] LNet: Removed LNI 192.168.203.28@tcp231 [ 1626.080634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.151753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1626.178495] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/0] [ 1626.183346] LNet: 83334:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1626.465323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1626.490841] LNet: Removed LNI 192.168.203.28@tcp231 [ 1626.518100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1626.563031] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/0] [ 1627.889378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.910744] LNet: Removed LNI 192.168.203.28@tcp231 [ 1627.930929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.054142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1628.082889] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/60] [ 1628.094366] LNet: 83485:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1628.614974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1628.659145] LNet: Removed LNI 192.168.203.28@tcp231 [ 1628.691721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1628.761116] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/60] [ 1630.276729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.295628] LNet: Removed LNI 192.168.203.28@tcp231 [ 1630.313478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.405342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1630.438359] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/180] [ 1630.457228] LNet: 83636:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1631.069903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1631.135277] LNet: Removed LNI 192.168.203.28@tcp231 [ 1631.171651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1631.220622] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/180] [ 1641.184658] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 04:20:45 (1787386845) [ 1641.770291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.797281] LNet: Removed LNI 192.168.203.28@tcp231 [ 1641.819757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.898901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1641.932167] LNet: Added LNI 192.168.203.28@tcp232 [8/256/0/180] [ 1642.877602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1642.931890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1642.998427] LNet: Removed LNI 192.168.203.28@tcp232 [ 1643.025631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1643.056586] LNet: Added LNI 192.168.203.28@tcp232 [8/256/0/180] [ 1643.812273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1643.853445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1643.923917] LNet: Removed LNI 192.168.203.28@tcp232 [ 1643.948141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1644.001112] LNet: Added LNI 192.168.203.28@tcp232 [8/256/0/180] [ 1644.927602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1644.966329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1645.016415] LNet: Removed LNI 192.168.203.28@tcp232 [ 1645.032940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1645.069941] LNet: Added LNI 192.168.203.28@tcp232 [8/256/0/180] [ 1645.833489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1645.863968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1645.894988] LNet: Removed LNI 192.168.203.28@tcp232 [ 1645.911198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1654.974035] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 04:20:59 (1787386859) [ 1655.728232] LNet: 85475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.747322] LNet: 85475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 1656.866542] Key type .llcrypt unregistered [ 1656.868423] Key type ._llcrypt unregistered [ 1664.505843] Key type ._llcrypt registered [ 1664.507961] Key type .llcrypt registered [ 1664.571885] LNet: 85704:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1664.571885] mask usage: [+|-] ... [ 1664.751758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1664.767364] alg: No test for adler32 (adler32-zlib) [ 1665.885338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1665.944908] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1679.330278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1692.837868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.893491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1692.934335] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1695.155932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_interrupt -r 2 [ 1695.239300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_interrupt -r 2 [ 1695.298457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_interrupt -r 2 [ 1695.355608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_interrupt -r 2 [ 1695.391685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_interrupt -r 2 [ 1695.424460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_interrupt -r 2 [ 1695.485723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_interrupt -r 2 [ 1695.530144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_interrupt -r 2 [ 1695.589713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp 192.168.203.128@tcp [ 1695.624461] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1695.634550] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1695.980279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1696.017903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1696.048030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1696.091237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1696.131977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1696.180967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1696.214362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1696.258392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1696.277940] LNet: 86139:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1696.293597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1696.323298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1696.361909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1696.398174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1696.472275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1696.556813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1697.207543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_dropped -r 2 [ 1697.241159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_dropped -r 2 [ 1697.286790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_dropped -r 2 [ 1697.324297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_dropped -r 2 [ 1697.387240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_dropped -r 2 [ 1697.430419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_dropped -r 2 [ 1697.485972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_dropped -r 2 [ 1697.530204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_dropped -r 2 [ 1697.568679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp 192.168.203.128@tcp [ 1697.604681] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1697.859254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1697.888714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1697.913864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1697.940813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1697.990859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1698.037858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1698.098436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1698.158236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1698.200982] LNet: 86259:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1698.234461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1698.288299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1698.323328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1698.351720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1698.404643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1698.453164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1699.086932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_aborted -r 2 [ 1699.136430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_aborted -r 2 [ 1699.183462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_aborted -r 2 [ 1699.239326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_aborted -r 2 [ 1699.307370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_aborted -r 2 [ 1699.378285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_aborted -r 2 [ 1699.452629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_aborted -r 2 [ 1699.510437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_aborted -r 2 [ 1699.586414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp 192.168.203.128@tcp [ 1699.627604] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1699.940852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1699.981667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1700.019376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1700.060261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1700.099229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1700.133340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1700.167791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1700.197463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1700.215781] LNet: 86380:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1700.235601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1700.260413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1700.309175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1700.337773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1700.377490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1700.413465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1701.090859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_no_route -r 2 [ 1701.130701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_no_route -r 2 [ 1701.175339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_no_route -r 2 [ 1701.225403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_no_route -r 2 [ 1701.278150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_no_route -r 2 [ 1701.331924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_no_route -r 2 [ 1701.386604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_no_route -r 2 [ 1701.436516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_no_route -r 2 [ 1701.495150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp 192.168.203.128@tcp [ 1702.033782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1702.070243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1702.092913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1702.118264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1702.156707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1702.225475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1702.282865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1702.331718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1702.354847] LNet: 86500:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1702.388400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1702.414131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1702.434396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1702.458765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1702.476480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1702.498770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1703.120238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_timeout -r 2 [ 1703.206654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_timeout -r 2 [ 1703.307109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_timeout -r 2 [ 1703.381888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_timeout -r 2 [ 1703.462263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_timeout -r 2 [ 1703.505366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_timeout -r 2 [ 1703.529460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_timeout -r 2 [ 1703.574690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_timeout -r 2 [ 1703.619694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp 192.168.203.128@tcp [ 1703.643987] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1703.649050] LNet: Skipped 1 previous similar message [ 1703.850173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1703.882392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1703.909365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1703.932228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1703.963144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1703.998880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1704.066879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1704.094897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1704.135755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1704.178814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1704.210702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1704.281667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1704.319589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1704.366827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1705.154742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e remote_dropped -r 2 [ 1705.218305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e remote_dropped -r 2 [ 1705.275800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e remote_dropped -r 2 [ 1705.354391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e remote_dropped -r 2 [ 1705.426852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e remote_dropped -r 2 [ 1705.464313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e remote_dropped -r 2 [ 1705.494842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e remote_dropped -r 2 [ 1705.526708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e remote_dropped -r 2 [ 1705.572437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp 192.168.203.128@tcp [ 1705.614301] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.128@tcp [ 1705.878722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1705.918282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1705.951118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1705.992649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1706.026951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1706.066442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1706.133569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1706.186607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1706.229917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1706.263228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1706.295659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1706.330575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1706.362456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1706.406674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1713.328358] LNet: 86815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.348924] LNetError: 86815:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.364808] LNet: Removed LNI 192.168.203.28@tcp [ 1713.403492] LNet: Removed LNI 192.168.203.28@tcp1 [ 1714.269666] Key type .llcrypt unregistered [ 1714.271811] Key type ._llcrypt unregistered [ 1736.474892] Key type ._llcrypt registered [ 1736.478246] Key type .llcrypt registered [ 1737.613281] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 04:22:22 (1787386942) [ 1738.073843] LNet: 87620:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1738.073843] mask usage: [+|-] ... [ 1738.242248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1738.275720] alg: No test for adler32 (adler32-zlib) [ 1739.255547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.322542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1739.361793] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1740.441182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1740.493559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.28@tcp [ 1740.533436] LNetError: 87792:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.203.28@tcp to recovery [ 1740.685739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.28@tcp [ 1740.861548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1748.405520] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 04:22:32 (1787386952) [ 1748.797653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.822608] LNet: 88382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1748.829651] LNetError: 88382:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1748.845949] LNet: Removed LNI 192.168.203.28@tcp [ 1748.859947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.936861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1748.967490] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1749.776085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1749.804546] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1750.917371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.28@tcp [ 1750.962843] LNetError: 88648:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.203.28@tcp to recovery [ 1751.968288] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1751.974702] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1752.247330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.28@tcp [ 1761.227399] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 04:22:46 (1787386966) [ 1763.714379] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg328-server [ 1765.594873] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 04:22:50 (1787386970) [ 1767.802752] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg328-server [ 1769.536804] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 04:22:54 (1787386974) [ 1771.538344] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg328-server [ 1773.656385] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 04:22:58 (1787386978) [ 1773.900205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.946076] LNet: 90279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1773.961210] LNetError: 90279:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1773.977342] LNet: Removed LNI 192.168.203.28@tcp [ 1773.987274] LNet: Removed LNI 192.168.203.28@tcp1 [ 1774.021652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.074772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1774.101678] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1775.008030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1775.108891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1775.176889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1775.318734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1775.383824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1775.459867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1783.841514] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 04:23:08 (1787386988) [ 1783.923156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.973584] LNet: 90998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.982731] LNetError: 90998:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1784.001461] LNetError: 90998:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1784.039855] LNet: Removed LNI 192.168.203.28@tcp [ 1784.060376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.202640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1784.227936] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1793.759024] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 04:23:18 (1787386998) [ 1793.879935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.912280] LNet: 91701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1793.921845] LNetError: 91701:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1793.944994] LNet: Removed LNI 192.168.203.28@tcp [ 1793.994129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.130306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1794.153213] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1804.374425] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 04:23:28 (1787387008) [ 1805.802509] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1807.331717] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 04:23:32 (1787387012) [ 1808.066927] LNet: 92744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.069340] LNetError: 92744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.087193] LNet: Removed LNI 192.168.203.28@tcp [ 1808.761293] Key type .llcrypt unregistered [ 1808.765988] Key type ._llcrypt unregistered [ 1815.633469] Key type ._llcrypt registered [ 1815.635657] Key type .llcrypt registered [ 1815.670310] LNet: 92974:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1815.670310] mask usage: [+|-] ... [ 1815.902493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1815.926921] alg: No test for adler32 (adler32-zlib) [ 1816.885164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1816.909245] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1824.583957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1829.756652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.128@tcp [ 1830.165925] LNet: 93163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.169201] LNetError: 93163:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1830.180744] LNet: Removed LNI 192.168.203.28@tcp [ 1830.537414] Key type .llcrypt unregistered [ 1830.539518] Key type ._llcrypt unregistered [ 1842.326817] Key type ._llcrypt registered [ 1842.331167] Key type .llcrypt registered [ 1843.721476] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1845.246603] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1847.316442] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 04:24:11 (1787387051) [ 1849.232575] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1851.217233] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 04:24:15 (1787387055) [ 1853.127565] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg328-server [ 1855.225325] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 04:24:19 (1787387059) [ 1857.153857] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg328-server [ 1859.060284] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 04:24:23 (1787387063) [ 1860.346178] LNet: 95215:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1860.346178] mask usage: [+|-] ... [ 1860.844675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1861.024689] alg: No test for adler32 (adler32-zlib) [ 1862.155144] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 1862.428147] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1864.160336] Key type lgssc registered [ 1865.530893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1886.789904] Key type lgssc unregistered [ 1887.104475] LNet: 96338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1887.114564] LNetError: 96338:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1887.140782] LNet: Removed LNI 192.168.203.28@tcp [ 1887.970760] Key type .llcrypt unregistered [ 1887.977681] Key type ._llcrypt unregistered [ 1904.327904] Key type ._llcrypt registered [ 1904.329670] Key type .llcrypt registered [ 1906.251657] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 04:25:10 (1787387110) [ 1906.515953] LNet: 97137:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1906.515953] mask usage: [+|-] ... [ 1906.692178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1906.716348] alg: No test for adler32 (adler32-zlib) [ 1907.747384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1907.896785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1907.981559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1908.073278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1908.169389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1915.665400] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 04:25:20 (1787387120) [ 1915.772759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1915.822398] LNet: 97747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.865150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.003302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1916.027191] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1917.187193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1917.214301] LNet: 97899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.232420] LNetError: 97899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1917.269170] LNet: Removed LNI 192.168.203.28@tcp [ 1917.296692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1917.361485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1917.397121] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1918.002493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1918.027688] LNetError: 97927:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1918.049687] LNet: Removed LNI 192.168.203.28@tcp [ 1918.083884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.152576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1918.189592] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1926.260734] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 04:25:30 (1787387130) [ 1926.509446] LNet: 98518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1926.521116] LNet: 98518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1926.541995] LNetError: 98518:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1926.566513] LNet: Removed LNI 192.168.203.28@tcp [ 1927.365606] Key type .llcrypt unregistered [ 1927.367451] Key type ._llcrypt unregistered [ 1934.282585] Key type ._llcrypt registered [ 1934.290484] Key type .llcrypt registered [ 1934.340148] LNet: 98751:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1934.340148] mask usage: [+|-] ... [ 1934.543382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1934.577878] alg: No test for adler32 (adler32-zlib) [ 1935.609305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1935.779809] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1935.786281] LNet: *** cfs_fail_loc=e003, val=100*** [ 1936.543410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1936.559952] LNet: *** cfs_fail_loc=e003, val=100*** [ 1936.563284] LNet: Skipped 2 previous similar messages [ 1936.569319] Ping reply received with mapping: [ 1936.572868] found metadata reply NI[192.168.203.28@tcp] [ 1936.576429] 1c cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1936.579607] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1936.584830] 5a 5a 5a 5a 5a 5a 5a 5a [ 1936.780562] LNet: 98918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1936.793471] LNetError: 98918:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1936.809991] LNet: Removed LNI 192.168.203.28@tcp [ 1937.566688] Key type .llcrypt unregistered [ 1937.575494] Key type ._llcrypt unregistered [ 1950.423567] Key type ._llcrypt registered [ 1950.428549] Key type .llcrypt registered [ 1952.193681] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 04:25:56 (1787387156) [ 1952.933625] Key type .llcrypt unregistered [ 1952.937566] Key type ._llcrypt unregistered [ 1961.047489] Key type ._llcrypt registered [ 1961.053056] Key type .llcrypt registered [ 1961.121714] LNet: 99914:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1961.121714] mask usage: [+|-] ... [ 1961.401293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1961.444145] alg: No test for adler32 (adler32-zlib) [ 1962.722574] LNetError: 99964:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1963.288758] Key type .llcrypt unregistered [ 1963.291222] Key type ._llcrypt unregistered [ 1968.982825] Key type ._llcrypt registered [ 1968.991500] Key type .llcrypt registered [ 1970.503554] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 04:26:14 (1787387174) [ 1971.200334] Key type .llcrypt unregistered [ 1971.202678] Key type ._llcrypt unregistered [ 1979.988756] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1979.997253] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1982.680701] Key type ._llcrypt registered [ 1982.686623] Key type .llcrypt registered [ 1982.753428] LNet: 100832:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1982.753428] mask usage: [+|-] ... [ 1982.956439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1982.972425] alg: No test for adler32 (adler32-zlib) [ 1983.948203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1984.113826] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1996.542499] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 04:26:41 (1787387201) [ 1996.706546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1996.731820] LNet: 101741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1996.745744] LNetError: 101741:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1996.788631] LNet: Removed LNI 10.1.2.3@tcp [ 1996.853823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1997.876528] Key type .llcrypt unregistered [ 1997.883596] Key type ._llcrypt unregistered [ 2006.840856] Key type ._llcrypt registered [ 2006.846960] Key type .llcrypt registered [ 2006.894867] LNet: 101977:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2006.894867] mask usage: [+|-] ... [ 2007.058193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2007.077634] alg: No test for adler32 (adler32-zlib) [ 2008.017357] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2008.094673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 2008.315437] LNet: 102034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2008.330640] LNetError: 102034:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.387313] LNet: Removed LNI 192.168.203.28@tcp [ 2010.163373] Key type .llcrypt unregistered [ 2010.165107] Key type ._llcrypt unregistered [ 2024.676879] Key type ._llcrypt registered [ 2024.678987] Key type .llcrypt registered [ 2026.606325] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 04:27:10 (1787387230) [ 2027.398689] Key type .llcrypt unregistered [ 2027.403831] Key type ._llcrypt unregistered [ 2037.717404] Key type ._llcrypt registered [ 2037.720275] Key type .llcrypt registered [ 2037.878367] LNet: 103019:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2037.878367] mask usage: [+|-] ... [ 2038.245459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2038.297534] alg: No test for adler32 (adler32-zlib) [ 2049.137733] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 04:27:33 (1787387253) [ 2049.281038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2049.361854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2049.517265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2049.553419] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2050.530845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2050.592933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2050.632261] LNet: Removed LNI 192.168.203.28@tcp [ 2051.012932] LNet: 103779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2051.534584] Key type .llcrypt unregistered [ 2051.538151] Key type ._llcrypt unregistered [ 2064.212344] Key type ._llcrypt registered [ 2064.218913] Key type .llcrypt registered [ 2066.195488] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 04:27:50 (1787387270) [ 2066.492303] LNet: 104540:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2066.492303] mask usage: [+|-] ... [ 2066.831864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2066.879162] alg: No test for adler32 (adler32-zlib) [ 2067.932733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2068.014858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2068.061736] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2069.239687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 2069.275372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.28@tcp [ 2069.294611] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2069.299625] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2069.302166] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2069, 0, 0 [ 2069.306883] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2069.310307] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2069.312562] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2069.319523] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2069, 0, 0 [ 2069.323716] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2069.326335] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2069.330191] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2069.333375] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2069, 0, 2119 [ 2069.340373] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2069.343374] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2069.347226] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2069.349648] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2069, 0, 2119 [ 2069.358279] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2076.179151] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 04:28:00 (1787387280) [ 2076.849237] LNet: 105319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.864829] LNetError: 105319:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2076.892601] LNet: Removed LNI 192.168.203.28@tcp [ 2077.937270] Key type .llcrypt unregistered [ 2077.941979] Key type ._llcrypt unregistered [ 2086.592955] Key type ._llcrypt registered [ 2086.600403] Key type .llcrypt registered [ 2086.691198] LNet: 105542:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2086.691198] mask usage: [+|-] ... [ 2087.143947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2087.190359] alg: No test for adler32 (adler32-zlib) [ 2088.143474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2088.195462] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2100.140843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 2113.748120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.857043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2113.894215] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 2115.410547] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2115.517426] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2115.520676] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2115.531744] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2118.058203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2118.083663] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2131.787209] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 04:28:56 (1787387336) [ 2131.944411] LNet: 106821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.965740] LNetError: 106821:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2131.971791] LNet: Removed LNI 192.168.203.28@tcp [ 2133.027146] LNet: Removed LNI 192.168.203.28@tcp1 [ 2133.037672] LNetError: 106821:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2133.053671] LNetError: 106821:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2133.102297] LNet: Removed LNI 10.1.2.3@tcp99 [ 2134.123458] Key type .llcrypt unregistered [ 2134.125972] Key type ._llcrypt unregistered [ 2143.159446] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2143.169903] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2145.091190] Key type ._llcrypt registered [ 2145.094115] Key type .llcrypt registered [ 2145.121918] LNet: 107087:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2145.121918] mask usage: [+|-] ... [ 2145.376762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2145.396736] alg: No test for adler32 (adler32-zlib) [ 2146.371355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.452633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2146.479699] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2147.364499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2147.401795] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2150.001823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2150.080137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2150.165255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2150.183914] LNetError: 107566:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2150.237729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2150.336312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2151.093393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2151.228647] LNet: 107586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2151.254293] LNetError: 107586:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2151.267645] LNet: Removed LNI 10.1.2.3@tcp [ 2152.295333] LNet: Removed LNI 192.168.203.28@tcp [ 2153.363759] Key type .llcrypt unregistered [ 2153.368795] Key type ._llcrypt unregistered [ 2168.425941] Key type ._llcrypt registered [ 2168.432317] Key type .llcrypt registered [ 2170.375149] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 04:29:34 (1787387374) [ 2170.698671] LNet: 108384:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2170.698671] mask usage: [+|-] ... [ 2171.005182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2171.055559] alg: No test for adler32 (adler32-zlib) [ 2172.169520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2172.226321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2172.264500] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2173.090056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 2173.196554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg328-client@tcp [ 2181.360577] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 04:29:45 (1787387385) [ 2181.473409] LNet: 109113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.484949] LNetError: 109113:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2181.516243] LNet: Removed LNI 192.168.203.28@tcp [ 2182.311159] Key type .llcrypt unregistered [ 2182.312887] Key type ._llcrypt unregistered [ 2189.381188] Key type ._llcrypt registered [ 2189.382796] Key type .llcrypt registered [ 2189.640436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2189.683899] alg: No test for adler32 (adler32-zlib) [ 2190.631719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2190.672299] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2190.708572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2198.792867] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 04:30:03 (1787387403) [ 2200.511436] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2202.539585] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 04:30:07 (1787387407) [ 2204.549362] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2206.943535] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 04:30:11 (1787387411) [ 2207.190617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2207.218938] LNet: 110541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2207.234573] LNetError: 110541:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2207.259558] LNet: Removed LNI 192.168.203.28@tcp [ 2207.310567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2207.389707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2207.404281] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2208.349240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2208.420501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2215.768575] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 04:30:20 (1787387420) [ 2216.080774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2216.103127] LNet: 111251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2216.123962] LNetError: 111251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2216.141736] LNet: Removed LNI 192.168.203.28@tcp [ 2216.172483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.260640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2216.314557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2217.043380] Key type .llcrypt unregistered [ 2217.047534] Key type ._llcrypt unregistered [ 2231.285947] Key type ._llcrypt registered [ 2231.288489] Key type .llcrypt registered [ 2232.989418] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 04:30:37 (1787387437) [ 2233.326800] LNet: 112058:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2233.326800] mask usage: [+|-] ... [ 2233.545741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2233.582546] alg: No test for adler32 (adler32-zlib) [ 2234.609488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2234.729505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2234.766320] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2235.710140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2235.787202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2243.692790] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 04:30:48 (1787387448) [ 2243.953389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2243.979711] LNet: 112808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2243.987040] LNetError: 112808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2245.033371] LNet: Removed LNI 192.168.203.28@tcp [ 2245.066422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2245.120222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2245.158704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2253.155222] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 04:30:57 (1787387457) [ 2253.445867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2253.507020] LNet: 113387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.560128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2253.692389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2254.397940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2254.436203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2254.582515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2255.129601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2255.190363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2255.261335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2255.278446] LNet: 113422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2255.326719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2255.389370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2255.885653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2255.953398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2256.039202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2256.567150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2256.607153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2264.056405] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 04:31:08 (1787387468) [ 2264.161825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2264.200156] LNet: 114018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2264.250684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2264.376255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2264.422303] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2265.237971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2265.252037] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2266.242214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2266.348102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2266.640967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2266.683749] LNetError: 114303:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2266.695203] LNet: Removed LNI 192.168.203.28@tcp [ 2266.714868] LNet: Removed LNI 192.168.203.28@tcp2 [ 2266.750665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2266.819647] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2266.826141] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2266.832314] LNet: 114309:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2267.411487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2267.699280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2267.748019] LNetError: 114346:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2267.756689] LNetError: 114346:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2267.767316] LNet: Removed LNI 192.168.203.28@tcp [ 2267.778589] LNet: Removed LNI 192.168.203.28@tcp2 [ 2267.838364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2267.938868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2267.996920] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2269.172248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2269.195834] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2270.092878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2270.178093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2270.244446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2270.573616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.595951] LNet: 114631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2270.608734] LNet: 114631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2270.628789] LNetError: 114631:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2270.646091] LNetError: 114631:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2270.668145] LNet: Removed LNI 192.168.203.28@tcp [ 2271.718735] LNet: Removed LNI 192.168.203.28@tcp2 [ 2271.766866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.908060] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2271.919834] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2271.931148] LNet: 114637:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2272.533433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2272.875061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2272.924868] LNet: Removed LNI 192.168.203.28@tcp [ 2273.986509] LNet: Removed LNI 192.168.203.28@tcp2 [ 2274.024432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2274.117745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2274.155950] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2275.058671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2275.109656] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2275.951729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2276.043126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2276.433229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2276.471732] LNetError: 114954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.489192] LNetError: 114954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2276.507876] LNet: Removed LNI 192.168.203.28@tcp [ 2276.529503] LNet: Removed LNI 192.168.203.28@tcp2 [ 2276.580703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2276.706473] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2276.728463] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2276.742433] LNet: 114960:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2277.442447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2277.689937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2278.752820] LNet: Removed LNI 192.168.203.28@tcp [ 2278.769179] LNet: Removed LNI 192.168.203.28@tcp2 [ 2278.828394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2278.934862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2279.001267] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2280.053719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2280.080232] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2281.149582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2281.191554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2281.285147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2281.579630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2281.595988] LNet: 115282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.602213] LNet: 115282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2282.656756] LNet: Removed LNI 192.168.203.28@tcp [ 2283.753121] LNet: Removed LNI 192.168.203.28@tcp2 [ 2283.796857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2283.893900] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2283.902887] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2283.906955] LNet: 115288:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2284.725034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2285.021778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2285.049799] LNetError: 115326:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2285.062098] LNetError: 115326:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2285.068115] LNet: Removed LNI 192.168.203.28@tcp [ 2286.120294] LNet: Removed LNI 192.168.203.28@tcp2 [ 2286.180750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2286.261159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2286.279295] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2287.022113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2287.053791] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2288.360334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2288.427463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2288.503759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2288.958501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2290.016372] LNet: Removed LNI 192.168.203.28@tcp [ 2290.030800] LNet: Removed LNI 192.168.203.28@tcp2 [ 2290.078254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2290.207979] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2290.217047] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2290.234803] LNet: 115629:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2291.036249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2291.498630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2292.576205] LNet: Removed LNI 192.168.203.28@tcp [ 2292.612494] LNet: Removed LNI 192.168.203.28@tcp2 [ 2292.663686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2292.757099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2292.787597] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2293.768269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2293.792867] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2294.919601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2294.982490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2295.024442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2295.113507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2295.679558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2296.736260] LNet: Removed LNI 192.168.203.28@tcp [ 2297.767724] LNet: Removed LNI 192.168.203.28@tcp2 [ 2297.814504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2297.912039] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2297.921888] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2298.665710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2307.320617] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2309.126608] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 04:31:53 (1787387513) [ 2309.366444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2309.420669] LNet: 116748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.435623] LNet: 116748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2309.441322] LNetError: 116748:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.446685] LNetError: 116748:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2309.452303] LNet: Removed LNI 192.168.203.28@tcp [ 2310.506931] LNet: Removed LNI 192.168.203.28@tcp2 [ 2310.545426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.951988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2310.978949] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2311.664735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2311.678854] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2312.743924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2312.769241] LNet: Removed LNI 10.1.2.3@tcp [ 2312.777080] LNet: Removed LNI 192.168.203.28@tcp [ 2312.801968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.880474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2312.928055] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2312.934060] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2313.347562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2313.371935] LNet: Removed LNI 192.168.203.28@tcp [ 2313.391427] LNet: Removed LNI 10.1.2.3@tcp [ 2320.489858] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 04:32:05 (1787387525) [ 2320.621368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2320.689228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2320.861222] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2321.037486] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2321.040849] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2321.045366] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2323.425734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2323.451279] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2324.240462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2324.272608] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2326.752934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2326.804499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2326.924365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.28@tcp -m PUT -e local_timeout -r 1 [ 2327.041660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2327.135219] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2327.140343] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2327.455764] LNetError: 117688:0:(lib-move.c:4714:lnet_parse()) 192.168.203.28@tcp, src 192.168.203.28@tcp: Dropping PUT (error -108 looking up sender) [ 2328.481248] LNet: Removed LNI 10.1.2.3@tcp [ 2329.513437] LNet: Removed LNI 192.168.203.28@tcp [ 2330.027698] Key type .llcrypt unregistered [ 2330.033130] Key type ._llcrypt unregistered [ 2344.194351] Key type ._llcrypt registered [ 2344.196661] Key type .llcrypt registered [ 2345.850448] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 04:32:30 (1787387550) [ 2346.131986] LNet: 118916:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2346.131986] mask usage: [+|-] ... [ 2346.406594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2346.434721] alg: No test for adler32 (adler32-zlib) [ 2347.457949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.797189] LNet: 119001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2348.576481] Key type .llcrypt unregistered [ 2348.584231] Key type ._llcrypt unregistered [ 2355.647613] Key type ._llcrypt registered [ 2355.658355] Key type .llcrypt registered [ 2355.698185] LNet: 119230:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2355.698185] mask usage: [+|-] ... [ 2355.894128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2355.915396] alg: No test for adler32 (adler32-zlib) [ 2356.860182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2356.883711] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2366.149905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 2366.439962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2367.118258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.128@tcp [ 2367.377534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2367.406897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2367.460923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2367.511711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2367.543543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2367.615818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2367.690793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2367.746694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2367.767601] LNet: 119452:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2367.785667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2367.815371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2367.846382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2367.875544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2367.899372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2367.945784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2368.703331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2368.952144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2368.983738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2369.016900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2369.061642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2369.113629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2369.174553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2369.191253] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.128@tcp [ 2369.232917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2369.279519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2369.306642] LNet: 119569:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2369.327973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2369.354444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2369.389371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2369.419572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2369.493731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2369.555988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2370.308448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.128@tcp [ 2370.591760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2370.653379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2370.688308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2370.748063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2370.792682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2370.816906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2370.843546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2370.877685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2370.895534] LNet: 119686:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2370.908982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2370.941776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2370.982740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2371.044992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2371.074670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2371.097631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2371.770336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2372.159629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2372.200304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2372.237714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2372.274040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2372.320973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2372.397489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2372.451172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2372.519644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2372.577454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2372.628405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2372.669392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2372.717741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2372.756361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2372.790179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2373.166319] LNet: 119850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2373.180391] LNetError: 119850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2373.193876] LNet: Removed LNI 192.168.203.28@tcp [ 2373.714666] Key type .llcrypt unregistered [ 2373.716430] Key type ._llcrypt unregistered [ 2394.133955] Key type ._llcrypt registered [ 2394.147100] Key type .llcrypt registered [ 2395.566385] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 04:33:20 (1787387600) [ 2396.021144] LNet: 120656:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2396.021144] mask usage: [+|-] ... [ 2396.342694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2396.374702] alg: No test for adler32 (adler32-zlib) [ 2397.278268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2397.632261] LNet: 120737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2398.198510] Key type .llcrypt unregistered [ 2398.200640] Key type ._llcrypt unregistered [ 2404.527797] Key type ._llcrypt registered [ 2404.530045] Key type .llcrypt registered [ 2404.557474] LNet: 120967:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2404.557474] mask usage: [+|-] ... [ 2404.647347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2404.656201] alg: No test for adler32 (adler32-zlib) [ 2405.558090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2405.588468] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2414.227829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 2423.853840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2423.937582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2423.991595] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 2425.001351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2425.680629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.128@tcp [ 2425.924434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2425.949139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2425.983827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2426.010922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2426.040331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2426.069791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2426.080465] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.128@tcp [ 2426.125402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2426.154187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2426.179675] LNet: 121391:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2426.223635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2426.274506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2426.323780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2426.376631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2426.465634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2426.537760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2427.361759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2427.588167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2427.644663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2427.685480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2427.741936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2427.815221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2427.847835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2427.891211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2427.924516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2427.945823] LNet: 121508:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2427.966425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2427.991500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2428.011187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2428.033937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2428.091293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2428.117436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2429.046245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.128@tcp [ 2429.396336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2429.456658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2429.532581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2429.619672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2429.717807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2429.795490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2429.834648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2429.908283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2429.954851] LNet: 121626:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2430.004690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2430.042124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2430.087866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2430.147016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2430.205238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2430.263243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2431.548509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2431.973226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2432.027693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2432.085420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2432.178104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2432.256424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2432.320545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2432.394443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2432.451193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2432.483580] LNet: 121743:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2432.517346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2432.564893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2432.607220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2432.650213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2432.686367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2432.741348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2439.896667] LNet: 121825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2439.926431] LNetError: 121825:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2439.942138] LNet: Removed LNI 192.168.203.28@tcp [ 2439.979426] LNet: Removed LNI 192.168.203.28@tcp1 [ 2440.916522] Key type .llcrypt unregistered [ 2440.924268] Key type ._llcrypt unregistered [ 2465.473421] Key type ._llcrypt registered [ 2465.478609] Key type .llcrypt registered [ 2466.908383] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 04:34:31 (1787387671) [ 2475.370729] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 04:34:39 (1787387679) [ 2482.464317] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 04:34:47 (1787387687) [ 2489.909442] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 04:34:54 (1787387694) [ 2498.625480] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 04:35:02 (1787387702) [ 2507.334361] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 04:35:11 (1787387711) [ 2516.383806] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 04:35:21 (1787387721) [ 2524.698215] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 04:35:29 (1787387729) [ 2533.640816] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 04:35:38 (1787387738) [ 2541.466581] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 04:35:46 (1787387746) [ 2548.221752] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 04:35:52 (1787387752) [ 2555.771785] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 04:36:00 (1787387760) [ 2563.609563] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 04:36:08 (1787387768) [ 2570.536697] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 04:36:15 (1787387775) [ 2576.806893] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 04:36:21 (1787387781) [ 2584.615540] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 04:36:29 (1787387789) [ 2594.333745] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 04:36:39 (1787387799) [ 2602.229575] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 04:36:46 (1787387806) [ 2602.464739] LNet: 132383:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2602.464739] mask usage: [+|-] ... [ 2602.631303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2602.658706] alg: No test for adler32 (adler32-zlib) [ 2603.555452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2603.659789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2603.675492] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2604.752667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2604.774338] LNet: 132556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2604.785605] LNetError: 132556:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2604.798166] LNet: Removed LNI 192.168.203.28@tcp [ 2604.821147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2604.866815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2604.887943] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2612.099278] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 04:36:57 (1787387817) [ 2619.871373] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 04:37:04 (1787387824) [ 2630.595656] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 04:37:14 (1787387834) [ 2639.463598] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 04:37:23 (1787387843) [ 2646.089853] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 04:37:30 (1787387850) [ 2653.056725] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 04:37:37 (1787387857) [ 2660.154451] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 04:37:45 (1787387865) [ 2667.673366] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 04:37:52 (1787387872) [ 2676.143591] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 04:38:00 (1787387880) [ 2683.173841] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 04:38:08 (1787387888) [ 2690.207764] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 04:38:15 (1787387895) [ 2697.706342] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 04:38:22 (1787387902) [ 2705.510369] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2480 sec ========== 04:38:29 (1787387909) [ 2705.738393] LNet: 140022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2705.743910] LNetError: 140022:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2705.754346] LNet: Removed LNI 192.168.203.28@tcp [ 2706.323326] Key type .llcrypt unregistered [ 2706.324964] Key type ._llcrypt unregistered [ 2721.193988] Key type ._llcrypt registered [ 2721.197484] Key type .llcrypt registered [ 2721.291312] LNet: 140497:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2721.291312] mask usage: [+|-] ... [ 2721.489883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2721.503237] alg: No test for adler32 (adler32-zlib) [ 2722.719447] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 2723.060235] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2724.793643] Key type lgssc registered [ 2726.015800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2790.073896] Lustre: Mounted lustre-client [ 2790.241253] LNet: 141971:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2790.241253] mask usage: [+|-] ... [ 2795.301412] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2804.463777] Lustre: Unmounted lustre-client [ 2828.467387] Key type lgssc unregistered [ 2828.701790] LNet: 142952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2828.715910] LNetError: 142952:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2828.739439] LNet: Removed LNI 192.168.203.28@tcp [ 2829.287160] Key type .llcrypt unregistered [ 2829.289655] Key type ._llcrypt unregistered