[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 409541470 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002375] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007014] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.009124] LSM: Security Framework initializing [ 0.010045] Yama: becoming mindful. [ 0.011027] SELinux: Initializing. [ 0.012060] *** VALIDATE selinux *** [ 0.019635] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023723] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025099] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026087] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027149] *** VALIDATE tmpfs *** [ 0.029008] *** VALIDATE proc *** [ 0.030265] *** VALIDATE cgroup *** [ 0.031006] *** VALIDATE cgroup2 *** [ 0.032237] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033123] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035032] Spectre V2 : User space: Vulnerable [ 0.036007] Speculative Store Bypass: Vulnerable [ 0.038902] debug: unmapping init [mem 0xffffffffaf459000-0xffffffffaf460fff] [ 0.040185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041650] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042016] ... version: 2 [ 0.043008] ... bit width: 48 [ 0.044008] ... generic registers: 4 [ 0.045011] ... value mask: 0000ffffffffffff [ 0.046009] ... max period: 00007fffffffffff [ 0.047012] ... fixed-purpose events: 3 [ 0.048008] ... event mask: 000000070000000f [ 0.049439] rcu: Hierarchical SRCU implementation. [ 0.051476] smp: Bringing up secondary CPUs ... [ 0.052426] x86: Booting SMP configuration: [ 0.053020] .... node #0, CPUs: #1 #2 #3 [ 0.061091] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064042] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.276024] node 0 deferred pages initialised in 210ms [ 0.279242] devtmpfs: initialized [ 0.280330] x86/mm: Memory block size: 128MB [ 0.282000] gcov: version magic: 0x41383552 [ 0.285210] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.288105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.290362] pinctrl core: initialized pinctrl subsystem [ 0.292190] [ 0.292699] ************************************************************* [ 0.295014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.297009] ** ** [ 0.299011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.300008] ** ** [ 0.302009] ** This means that this kernel is built to expose internal ** [ 0.304009] ** IOMMU data structures, which may compromise security on ** [ 0.306009] ** your system. ** [ 0.308010] ** ** [ 0.310008] ** If you see this message and you are not debugging the ** [ 0.311008] ** kernel, report this immediately to your vendor! ** [ 0.313008] ** ** [ 0.315009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.317018] ************************************************************* [ 0.319668] NET: Registered protocol family 16 [ 0.321408] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.323091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.326064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.330186] cpuidle: using governor menu [ 0.332601] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.335501] PCI: Using configuration type 1 for base access [ 0.337129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.345101] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.347036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.351004] cryptd: max_cpu_qlen set to 1000 [ 0.353186] ACPI: Added _OSI(Module Device) [ 0.354015] ACPI: Added _OSI(Processor Device) [ 0.356010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.357011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.363000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.368453] ACPI: Interpreter enabled [ 0.370051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.371007] ACPI: Using IOAPIC for interrupt routing [ 0.372135] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.374315] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.382021] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.383032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.386024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.390080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.395507] acpiphp: Slot [2] registered [ 0.397136] acpiphp: Slot [5] registered [ 0.398107] acpiphp: Slot [6] registered [ 0.400117] acpiphp: Slot [3] registered [ 0.402100] acpiphp: Slot [4] registered [ 0.404112] acpiphp: Slot [7] registered [ 0.405081] acpiphp: Slot [8] registered [ 0.407081] acpiphp: Slot [9] registered [ 0.408081] acpiphp: Slot [10] registered [ 0.410059] acpiphp: Slot [11] registered [ 0.412090] acpiphp: Slot [12] registered [ 0.414079] acpiphp: Slot [13] registered [ 0.415059] acpiphp: Slot [14] registered [ 0.417070] acpiphp: Slot [15] registered [ 0.419077] acpiphp: Slot [16] registered [ 0.420386] acpiphp: Slot [17] registered [ 0.423379] acpiphp: Slot [18] registered [ 0.425090] acpiphp: Slot [19] registered [ 0.427171] acpiphp: Slot [20] registered [ 0.428064] acpiphp: Slot [21] registered [ 0.429284] acpiphp: Slot [22] registered [ 0.430076] acpiphp: Slot [23] registered [ 0.432075] acpiphp: Slot [24] registered [ 0.433067] acpiphp: Slot [25] registered [ 0.434062] acpiphp: Slot [26] registered [ 0.435000] acpiphp: Slot [27] registered [ 0.435000] acpiphp: Slot [28] registered [ 0.436062] acpiphp: Slot [29] registered [ 0.437064] acpiphp: Slot [30] registered [ 0.439067] acpiphp: Slot [31] registered [ 0.440061] PCI host bridge to bus 0000:00 [ 0.441013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.443012] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.445014] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.448017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.450012] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.452016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.454222] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.456863] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.460262] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.467012] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.471054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.474013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.476013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.478009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.480554] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.482632] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.485048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.488887] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.492767] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.499013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.503011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.508409] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.517013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.523013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.535000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.545215] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.553013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.556952] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.567014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.575290] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.578386] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.581365] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.583373] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.586207] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.590119] iommu: Default domain type: Passthrough [ 0.592452] SCSI subsystem initialized [ 0.593166] ACPI: bus type USB registered [ 0.595105] usbcore: registered new interface driver usbfs [ 0.597077] usbcore: registered new interface driver hub [ 0.599059] usbcore: registered new device driver usb [ 0.601151] pps_core: LinuxPPS API ver. 1 registered [ 0.602006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.606183] PTP clock support registered [ 0.609041] EDAC MC: Ver: 3.0.0 [ 0.610339] PCI: Using ACPI for IRQ routing [ 0.611659] NetLabel: Initializing [ 0.613013] NetLabel: domain hash size = 128 [ 0.615008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.616069] NetLabel: unlabeled traffic allowed by default [ 0.619035] vgaarb: loaded [ 0.620264] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.622012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.628706] clocksource: Switched to clocksource kvm-clock [ 0.739110] VFS: Disk quotas dquot_6.6.0 [ 0.740237] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.742294] *** VALIDATE ramfs *** [ 0.743373] *** VALIDATE hugetlbfs *** [ 0.745201] pnp: PnP ACPI init [ 0.748079] pnp: PnP ACPI: found 6 devices [ 0.763615] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.767116] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.769422] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.771794] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.774304] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.776844] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.779844] NET: Registered protocol family 2 [ 0.782653] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.787056] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.790309] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.795730] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.798500] TCP: Hash tables configured (established 65536 bind 65536) [ 0.801351] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.804283] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.808635] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.811600] NET: Registered protocol family 1 [ 0.814306] RPC: Registered named UNIX socket transport module. [ 0.817035] RPC: Registered udp transport module. [ 0.818666] RPC: Registered tcp transport module. [ 0.819905] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.821330] NET: Registered protocol family 44 [ 0.822352] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.823711] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.825335] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.827099] PCI: CLS 0 bytes, default 64 [ 0.828510] Unpacking initramfs... [ 2.182623] debug: unmapping init [mem 0xffff9f46bcc64000-0xffff9f46bffcffff] [ 2.189274] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.191719] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.194766] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.658974] Initialise system trusted keyrings [ 2.660707] Key type blacklist registered [ 2.663235] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.671557] zbud: loaded [ 2.674083] *** VALIDATE nfs *** [ 2.675095] *** VALIDATE nfs4 *** [ 2.676531] pstore: using deflate compression [ 2.679152] Platform Keyring initialized [ 2.760621] NET: Registered protocol family 38 [ 2.761837] Key type asymmetric registered [ 2.763173] Asymmetric key parser 'x509' registered [ 2.764467] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.766503] io scheduler mq-deadline registered [ 2.768511] io scheduler kyber registered [ 2.769470] io scheduler bfq registered [ 2.770592] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.772782] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.775058] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.777011] ACPI: Power Button [PWRF] [ 2.781448] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.786847] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.797044] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.824618] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.852080] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.858174] Non-volatile memory driver v1.3 [ 2.860526] Linux agpgart interface v0.103 [ 2.888127] virtio_blk virtio1: [vda] 145248 512-byte logical blocks (74.4 MB/70.9 MiB) [ 2.891149] vda: detected capacity change from 0 to 74366976 [ 2.906706] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.909719] vdb: detected capacity change from 0 to 1073741824 [ 2.915326] libphy: Fixed MDIO Bus: probed [ 2.920546] usbcore: registered new interface driver usbserial_generic [ 2.922602] usbserial: USB Serial support registered for generic [ 2.924563] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.928306] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.929692] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.931504] mousedev: PS/2 mouse device common for all mice [ 2.935627] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.936398] rtc_cmos 00:05: RTC can wake from S4 [ 2.940762] rtc_cmos 00:05: registered as rtc0 [ 2.942263] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.943150] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.946544] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.949403] intel_pstate: CPU model not supported [ 2.956389] hid: raw HID events driver (C) Jiri Kosina [ 2.958538] usbcore: registered new interface driver usbhid [ 2.960447] usbhid: USB HID core driver [ 2.962088] drop_monitor: Initializing network drop monitor service [ 2.964552] Initializing XFRM netlink socket [ 2.966836] NET: Registered protocol family 10 [ 2.969663] Segment Routing with IPv6 [ 2.971259] NET: Registered protocol family 17 [ 2.974626] mpls_gso: MPLS GSO support [ 2.980841] RAS: Correctable Errors collector initialized. [ 2.983251] AVX version of gcm_enc/dec engaged. [ 2.984685] AES CTR mode by8 optimization enabled [ 3.041110] sched_clock: Marking stable (3041079554, 0)->(3852195750, -811116196) [ 3.044741] registered taskstats version 1 [ 3.046715] Loading compiled-in X.509 certificates [ 3.048698] zswap: loaded using pool lzo/zbud [ 3.070473] Key type big_key registered [ 3.080639] Key type encrypted registered [ 3.082127] ima: No TPM chip found, activating TPM-bypass! [ 3.083641] ima: Allocated hash algorithm: sha1 [ 3.085258] ima: No architecture policies found [ 3.086651] evm: Initialising EVM extended attributes: [ 3.087870] evm: security.selinux [ 3.088592] evm: security.ima [ 3.089401] evm: security.capability [ 3.090408] evm: HMAC attrs: 0x1 [ 3.091911] rtc_cmos 00:05: setting system clock to 2026-06-29 17:58:21 UTC (1782755901) [ 3.097263] debug: unmapping init [mem 0xffffffffb0403000-0xffffffffb05fffff] [ 3.099347] debug: unmapping init [mem 0xffffffffaf182000-0xffffffffaf458fff] [ 3.107068] Write protecting the kernel read-only data: 28672k [ 3.110114] debug: unmapping init [mem 0xffffffffad803000-0xffffffffad9fffff] [ 3.112871] debug: unmapping init [mem 0xffffffffae114000-0xffffffffae1fffff] [ 3.138879] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.145897] systemd[1]: Detected virtualization kvm. [ 3.147364] systemd[1]: Detected architecture x86-64. [ 3.148968] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.178591] systemd[1]: No hostname configured. [ 3.180433] systemd[1]: Set hostname to . [ 3.182442] random: systemd: uninitialized urandom read (16 bytes read) [ 3.184469] systemd[1]: Initializing machine ID from random generator. [ 3.226221] random: ln: uninitialized urandom read (6 bytes read) [ 3.305754] random: systemd: uninitialized urandom read (16 bytes read) [ 3.308187] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.312176] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.316322] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.920485] device-mapper: uevent: version 1.0.3 [ 3.922826] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [[ 4.525782] random: fast init done  OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.565860] virtio_net virtio0 ens2: renamed from eth0 [ 4.645556] scsi host0: ata_piix [ 4.659918] scsi host1: ata_piix [ 4.661543] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.664189] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.917982] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.593981] random: crng init done [ 9.595598] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.840122] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.902628] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.153045] SELinux: Disabled at runtime. [ 11.209557] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.218275] systemd[1]: Detected virtualization kvm. [ 11.220379] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.654572] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.658658] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.663106] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.667139] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.671204] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.677646] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.682215] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ 11.828710] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.065236] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.306103] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.327252] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.434201] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.449645] EDAC sbridge: Ver: 1.1.2 [ 13.333417] Key type dns_resolver registered [ 13.592158] NFS: Registering the id_resolver key type [ 13.593644] Key type id_resolver registered [ 13.594774] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg405-client login: [ 40.482228] hrtimer: interrupt took 10548080 ns [ 52.075722] libcfs: loading out-of-tree module taints kernel. [ 52.203272] Key type ._llcrypt registered [ 52.204864] Key type .llcrypt registered [ 52.606862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 52.617180] alg: No test for adler32 (adler32-zlib) [ 53.867403] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 54.466280] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 56.344238] Key type lgssc registered [ 57.775049] Lustre: Echo OBD driver; http://www.lustre.org/ [ 169.511987] Lustre: Mounted lustre-client [ 173.816764] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 184.195530] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing check_logdir /tmp/testlogs/ [ 188.152747] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing yml_node [ 191.992567] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 194.144425] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 195.040595] Lustre: lustre-OST0000-osc-ffff9f4712e87800: disconnect after 23s idle [ 196.403532] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 197.750407] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Jun 29 14:01:34 EDT 2026 [ 210.300902] Lustre: Unmounted lustre-client [ 228.528380] Key type lgssc unregistered [ 228.763959] LNet: 8395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 228.770385] LNetError: 8395:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 228.786285] LNet: Removed LNI 192.168.204.5@tcp [ 229.365684] Key type .llcrypt unregistered [ 229.369246] Key type ._llcrypt unregistered [ 248.387990] Key type ._llcrypt registered [ 248.390680] Key type .llcrypt registered [ 248.422444] LNet: 8973:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 248.422444] mask usage: [+|-] ... [ 248.529451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 248.543245] alg: No test for adler32 (adler32-zlib) [ 249.401666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 249.443812] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 249.460293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 252.248629] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 252.375521] LNet: 9277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 252.379792] LNetError: 9277:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 252.390905] LNet: Removed LNI 192.168.204.5@tcp [ 252.919463] Key type .llcrypt unregistered [ 252.921573] Key type ._llcrypt unregistered [ 257.540924] Key type ._llcrypt registered [ 257.543100] Key type .llcrypt registered [ 258.762820] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 14:02:35 (1782756155) [ 259.016769] LNet: 9647:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 259.016769] mask usage: [+|-] ... [ 259.177937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 259.207524] alg: No test for adler32 (adler32-zlib) [ 260.063432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.119594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 260.138581] LNet: 9688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 260.155929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.192576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 260.205032] LNet: 9695:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 266.250384] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 14:02:43 (1782756163) [ 266.432409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 266.447639] LNet: 10272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 266.475699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 266.561189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 272.861437] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 14:02:49 (1782756169) [ 273.257533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.283591] LNet: 10857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 273.321873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.398914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 279.122827] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 14:02:56 (1782756176) [ 279.320473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.352190] LNet: 11441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.375362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 279.443537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 284.928854] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 14:03:02 (1782756182) [ 285.238454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 285.258629] LNet: 12026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 285.280447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 285.330600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 285.655184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 291.460906] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 14:03:08 (1782756188) [ 291.652305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 291.689808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 291.759419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 292.103371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 292.683748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 298.843606] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 14:03:16 (1782756196) [ 299.042087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.069643] LNet: 13212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.074995] LNet: 13212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 299.094600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.150937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 304.738727] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 14:03:22 (1782756202) [ 304.893161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.942843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.998484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 305.042189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 305.285514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 305.311580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 305.698652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 305.745090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 306.130123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 306.165508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 306.559230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 306.592900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 307.062302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 307.105180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 307.498427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 307.526504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 307.814195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 307.842516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 308.289745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 308.322250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 308.684443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 308.707800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 309.063985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 309.111726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 309.401674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 309.432900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 309.814796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 309.847263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 315.035912] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 14:03:32 (1782756212) [ 315.116974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 315.142330] LNet: 14473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 315.148849] LNet: 14473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 315.166883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.222972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 315.255156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 319.758514] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 14:03:37 (1782756217) [ 319.831893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.882744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.957484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 319.989726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 324.610941] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 14:03:42 (1782756222) [ 324.737226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.790786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.884374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 324.915139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 329.371411] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 14:03:46 (1782756226) [ 329.467047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.505446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.593639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 329.626690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 334.390803] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 14:03:51 (1782756231) [ 334.462405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.492492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.583100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 334.645100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 339.842309] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 14:03:57 (1782756237) [ 339.944592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.985111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.091381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 340.159290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 346.142676] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 14:04:03 (1782756243) [ 346.293497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.350885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.455242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 346.860687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 352.707960] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 14:04:10 (1782756250) [ 352.791849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.820762] LNet: 18584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.838222] LNet: 18584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 352.867862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.077672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 353.325063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 358.823139] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 14:04:16 (1782756256) [ 359.087786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.123929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.239325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 359.411617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 364.339778] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 14:04:21 (1782756261) [ 364.572957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.612217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.028332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 365.278376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 371.062857] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 14:04:28 (1782756268) [ 371.206351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.248369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.298511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 371.347798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 376.499352] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 14:04:33 (1782756273) [ 376.707975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.739749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.802871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 376.856529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 382.165066] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 14:04:39 (1782756279) [ 382.233625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.279795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.337883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 387.262104] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 14:04:44 (1782756284) [ 387.317679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.362320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.417293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 391.915045] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 14:04:49 (1782756289) [ 392.049989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.077252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.121980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 392.403303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 396.895824] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 14:04:54 (1782756294) [ 396.963920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.001905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.068367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 397.100622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 401.363488] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 14:04:58 (1782756298) [ 401.486433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.526682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.582185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 401.609244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 405.969036] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 14:05:03 (1782756303) [ 406.122624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.145513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.210469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 406.237609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 410.661993] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 14:05:08 (1782756308) [ 410.836114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.881806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.952059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 410.987184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 415.135317] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 14:05:12 (1782756312) [ 415.288790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.324199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.362125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 415.411603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 415.429081] LNetError: 25792:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 415.469587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 415.499235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 419.958709] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 14:05:17 (1782756317) [ 420.029756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.047941] LNet: 26364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.051676] LNet: 26364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 420.077161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.159536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Qq8Cp8OCUl [ 420.235420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Qq8Cp8OCUl [ 420.277288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Qq8Cp8OCUl [ 425.039937] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 14:05:22 (1782756322) [ 425.242640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.280618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.373643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 425.407305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 425.440285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 429.782650] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 14:05:27 (1782756327) [ 429.961909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.995231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.056870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 431.617640] LNet: Message forwarding will be enabled in 2s [ 434.720398] LNet: Message forwarding enabled [ 434.947693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 434.967756] LNet: Message forwarding will stop in 2s to 6s [ 439.318987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 439.639637] LNet: Message forwarding disabled [ 440.656147] LNet: Message forwarding will be enabled in 2s [ 443.745625] LNet: Message forwarding enabled [ 445.794933] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 14:05:43 (1782756343) [ 445.923137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.878181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.936309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 451.373467] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 14:05:48 (1782756348) [ 451.560911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.587539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.638490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 451.668793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 451.698457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 451.736689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 451.770577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 451.794137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 451.817962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 451.848421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 451.880067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 451.903851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 451.935578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 451.966331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 451.990971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 452.017567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 452.049650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 452.084589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 452.124645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 452.154886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 452.186678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 452.209866] LNetError: 28862:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 456.689361] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 14:05:54 (1782756354) [ 456.862283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.893683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.944396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.859046] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 14:05:59 (1782756359) [ 463.210864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg405-server [ 464.496709] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 14:06:01 (1782756361) [ 464.644239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.673401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.723097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 464.745801] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 465.893472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[6-10]@tcp [ 465.913751] LNet: 30578:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 465.946416] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.6@tcp is being used as a gateway but routing feature is not turned on [ 465.953115] LNetError: 30368:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.6@tcp has gone from up to down [ 466.987716] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.6@tcp is being used as a gateway but routing feature is not turned on [ 467.002391] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 468.027743] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.6@tcp is being used as a gateway but routing feature is not turned on [ 468.048350] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 469.043753] LNetError: 30579:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.9@tcp has gone from up to down [ 469.063044] LNetError: 30579:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 470.087791] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.7@tcp is being used as a gateway but routing feature is not turned on [ 470.111786] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 474.203024] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.7@tcp is being used as a gateway but routing feature is not turned on [ 474.256238] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 11 previous similar messages [ 474.313264] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 14:06:09 (1782756369) [ 478.148850] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg405-server [ 481.903274] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 14:06:17 (1782756377) [ 482.213719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.336110] LNetError: 30367:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.204.6@tcp [ 483.233273] LNetError: 31513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 483.251631] LNet: Removed LNI 192.168.204.5@tcp [ 483.325422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.458890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 483.615190] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 484.610702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[6-11/2]@tcp [ 484.650944] LNet: 31631:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 484.666917] LNet: 31631:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 484.694363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[6-11/2]@tcp [ 484.716343] LNetError: 31518:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.10@tcp is being used as a gateway but routing feature is not turned on [ 484.766400] LNetError: 31518:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 23 previous similar messages [ 484.773725] LNetError: 31518:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.10@tcp has gone from up to down [ 484.806521] LNetError: 31518:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 494.679982] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 14:06:30 (1782756390) [ 494.873781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.923530] LNetError: 32219:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 494.956692] LNet: Removed LNI 192.168.204.5@tcp [ 495.015716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.171243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 495.217286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.241574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.332967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.424984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.499965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.537334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 495.599604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.625549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.686638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.758217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.826801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.875638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 495.956777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.988936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.051168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 496.218442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.291050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.493211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 496.562294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.595737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.732352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 496.826458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.862857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.928106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 497.084972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.170174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.238963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 497.355782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.476510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.622374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 497.719303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.768567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.900395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 508.207121] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 14:06:44 (1782756404) [ 511.232304] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg405-server [ 514.231728] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 14:06:50 (1782756410) [ 516.936926] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg405-server [ 520.266624] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 14:06:56 (1782756416) [ 521.514848] Key type .llcrypt unregistered [ 521.516695] Key type ._llcrypt unregistered [ 528.905421] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 529.040778] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 529.045937] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 529.055757] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 531.695843] Key type ._llcrypt registered [ 531.697733] Key type .llcrypt registered [ 531.753324] LNet: 33908:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 531.753324] mask usage: [+|-] ... [ 531.884157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 531.902557] alg: No test for adler32 (adler32-zlib) [ 532.873183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.953862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 532.988967] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 533.763605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 533.783514] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 535.976458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 536.007236] LNet: Removed LNI 10.1.2.3@tcp [ 536.188726] LNet: 34350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 536.195528] LNetError: 34350:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 536.219823] LNet: Removed LNI 192.168.204.5@tcp [ 536.855639] Key type .llcrypt unregistered [ 536.858140] Key type ._llcrypt unregistered [ 551.210448] Key type ._llcrypt registered [ 551.215681] Key type .llcrypt registered [ 552.679792] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 14:07:29 (1782756449) [ 553.116397] Key type .llcrypt unregistered [ 553.118350] Key type ._llcrypt unregistered [ 559.456780] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 559.594752] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 559.601204] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 559.613154] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 562.372847] Key type ._llcrypt registered [ 562.376153] Key type .llcrypt registered [ 562.431576] LNet: 35388:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 562.431576] mask usage: [+|-] ... [ 562.660156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 562.700386] alg: No test for adler32 (adler32-zlib) [ 563.653610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.716664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 563.761109] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 564.646834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 564.670484] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 567.117301] LNet: 35846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.126145] LNetError: 35846:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 567.136818] LNet: Removed LNI 10.1.2.3@tcp [ 567.152187] LNet: Removed LNI 192.168.204.5@tcp [ 567.811422] Key type .llcrypt unregistered [ 567.816299] Key type ._llcrypt unregistered [ 579.271837] Key type ._llcrypt registered [ 579.275853] Key type .llcrypt registered [ 580.674327] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 14:07:57 (1782756477) [ 581.211254] Key type .llcrypt unregistered [ 581.213972] Key type ._llcrypt unregistered [ 587.809838] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 587.982564] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 587.985787] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 587.989635] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 590.627575] Key type ._llcrypt registered [ 590.631559] Key type .llcrypt registered [ 590.662399] LNet: 36880:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 590.662399] mask usage: [+|-] ... [ 590.802562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 590.839358] alg: No test for adler32 (adler32-zlib) [ 591.743197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.783360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 591.801885] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 593.879297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 593.903596] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 596.146417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 596.168276] LNet: Removed LNI 10.1.2.3@tcp [ 596.181677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 596.219651] LNet: Removed LNI 10.1.2.31@tcp [ 596.231481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 596.270041] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 597.011306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 597.037934] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 598.017986] LNet: 37764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 598.031689] LNetError: 37764:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 598.039325] LNet: Removed LNI 10.1.2.31@tcp [ 598.053021] LNet: Removed LNI 10.1.2.3@tcp [ 598.748441] Key type .llcrypt unregistered [ 598.753185] Key type ._llcrypt unregistered [ 611.880423] Key type ._llcrypt registered [ 611.883789] Key type .llcrypt registered [ 613.428715] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 14:08:30 (1782756510) [ 613.698492] LNet: 38556:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 613.698492] mask usage: [+|-] ... [ 613.832318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 613.852609] alg: No test for adler32 (adler32-zlib) [ 614.767914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.809694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 614.826752] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 615.599407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.624478] LNet: 38716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.633702] LNetError: 38716:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.649802] LNet: Removed LNI 192.168.204.5@tcp [ 615.670633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.734404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.5@tcp [ 615.767723] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 616.607902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.5@tcp [ 616.648439] LNet: Removed LNI 192.168.204.5@tcp [ 616.723214] LNet: 38852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.448578] Key type .llcrypt unregistered [ 617.452684] Key type ._llcrypt unregistered [ 630.756440] Key type ._llcrypt registered [ 630.758105] Key type .llcrypt registered [ 632.331937] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 14:08:48 (1782756528) [ 634.494718] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg405-server [ 637.150355] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 14:08:53 (1782756533) [ 638.095347] Key type .llcrypt unregistered [ 638.097176] Key type ._llcrypt unregistered [ 646.398714] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 646.615114] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 646.624146] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 646.630410] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 649.205535] Key type ._llcrypt registered [ 649.213779] Key type .llcrypt registered [ 649.283484] LNet: 40229:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 649.283484] mask usage: [+|-] ... [ 649.543611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 649.566971] alg: No test for adler32 (adler32-zlib) [ 650.527923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.600310] LNet: Added LNI 192.168.204.5@tcp1 [8/256/0/180] [ 651.392172] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 654.176773] LNet: 40689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.191217] LNetError: 40689:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.206343] LNet: Removed LNI 192.168.204.5@tcp1 [ 654.230277] LNet: Removed LNI 10.1.2.3@tcp2 [ 655.115325] Key type .llcrypt unregistered [ 655.120593] Key type ._llcrypt unregistered [ 668.322495] Key type ._llcrypt registered [ 668.325522] Key type .llcrypt registered [ 669.294259] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 14:09:26 (1782756566) [ 669.878991] LNet: 41493:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 669.878991] mask usage: [+|-] ... [ 670.050040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 670.067586] alg: No test for adler32 (adler32-zlib) [ 671.050911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.094460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 671.120590] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 671.934442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.973557] LNet: 41658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.982694] LNetError: 41658:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.005043] LNet: Removed LNI 192.168.204.5@tcp [ 672.051213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.122703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 672.160268] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 680.013322] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 14:09:36 (1782756576) [ 680.360692] LNet: 42253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.371259] LNetError: 42253:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 680.394219] LNet: Removed LNI 192.168.204.5@tcp [ 681.369403] Key type .llcrypt unregistered [ 681.373200] Key type ._llcrypt unregistered [ 687.767450] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 687.903219] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 687.910582] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 687.914606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 690.619911] Key type ._llcrypt registered [ 690.623654] Key type .llcrypt registered [ 690.694674] LNet: 42528:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 690.694674] mask usage: [+|-] ... [ 690.875898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 690.902467] alg: No test for adler32 (adler32-zlib) [ 691.849276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.901825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 691.920538] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 692.781838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 692.797910] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 694.931385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.964424] LNet: 42994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.974129] LNetError: 42994:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 694.985218] LNet: Removed LNI 10.1.2.3@tcp [ 695.004371] LNet: Removed LNI 192.168.204.5@tcp [ 695.050340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.137264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 695.173085] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 695.185135] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 695.925366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.944314] LNet: 43021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.950637] LNetError: 43021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 695.963943] LNetError: 43021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 695.974676] LNet: Removed LNI 10.1.2.3@tcp [ 695.986767] LNet: Removed LNI 192.168.204.5@tcp [ 696.025636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.088511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 696.106382] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 696.111923] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 696.677214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 696.705080] LNet: Added LNI 192.168.204.5@tcp2 [8/256/0/180] [ 697.878396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.915253] LNet: 43183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.924317] LNetError: 43183:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 697.933533] LNetError: 43183:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 697.945745] LNet: Removed LNI 10.1.2.3@tcp [ 697.954933] LNet: Removed LNI 192.168.204.5@tcp [ 697.971781] LNet: Removed LNI 192.168.204.5@tcp2 [ 698.012381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.091238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 698.151078] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 698.167308] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 698.177818] LNet: Added LNI 192.168.204.5@tcp2 [8/256/0/180] [ 706.332715] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 14:10:03 (1782756603) [ 706.431950] LNet: 43772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.444126] LNetError: 43772:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 706.451683] LNetError: 43772:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 706.459133] LNet: Removed LNI 10.1.2.3@tcp [ 706.467780] LNet: Removed LNI 192.168.204.5@tcp [ 706.503892] LNet: Removed LNI 192.168.204.5@tcp2 [ 707.352122] Key type .llcrypt unregistered [ 707.358273] Key type ._llcrypt unregistered [ 714.461863] Key type ._llcrypt registered [ 714.468526] Key type .llcrypt registered [ 714.543508] LNet: 44007:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 714.543508] mask usage: [+|-] ... [ 714.717529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 714.745763] alg: No test for adler32 (adler32-zlib) [ 715.694644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.788721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 715.837277] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 716.614807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 716.646495] LNet: 44176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.656693] LNetError: 44176:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 716.680668] LNet: Removed LNI 192.168.204.5@tcp [ 716.717959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.796762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 716.826318] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 717.314104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.336812] LNet: 44206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.355062] LNetError: 44206:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 717.371498] LNet: Removed LNI 192.168.204.5@tcp [ 717.402890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.475103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 717.504955] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 717.935022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.984667] LNet: Removed LNI 192.168.204.5@tcp [ 718.005401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.085412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 718.129505] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 718.769174] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 718.836974] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 718.840791] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 718.850847] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 721.344698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.382024] LNet: 44314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.394630] LNet: 44314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 721.409906] LNetError: 44314:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.415972] LNetError: 44314:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 721.433216] LNet: Removed LNI 192.168.204.5@tcp [ 721.455470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.530779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 721.551439] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 729.316205] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 14:10:25 (1782756625) [ 729.420850] LNet: 44903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.431124] LNetError: 44903:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.470684] LNet: Removed LNI 192.168.204.5@tcp [ 730.254988] Key type .llcrypt unregistered [ 730.258068] Key type ._llcrypt unregistered [ 737.801681] Key type ._llcrypt registered [ 737.809320] Key type .llcrypt registered [ 737.838319] LNet: 45137:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 737.838319] mask usage: [+|-] ... [ 737.995841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 738.011361] alg: No test for adler32 (adler32-zlib) [ 738.918613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.996373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 739.032197] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 740.005103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.024226] LNet: 45311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.030725] LNetError: 45311:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 740.047700] LNet: Removed LNI 192.168.204.5@tcp [ 740.076601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.154761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 740.194585] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 747.027657] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 14:10:44 (1782756644) [ 747.316199] LNet: 45903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 747.328138] LNetError: 45903:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 747.362907] LNet: Removed LNI 192.168.204.5@tcp [ 748.052721] Key type .llcrypt unregistered [ 748.056687] Key type ._llcrypt unregistered [ 753.925520] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 754.061681] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 754.071289] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 754.073905] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 756.629406] Key type ._llcrypt registered [ 756.631224] Key type .llcrypt registered [ 756.656540] LNet: 46179:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 756.656540] mask usage: [+|-] ... [ 756.794934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 756.818942] alg: No test for adler32 (adler32-zlib) [ 757.765205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.821326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 757.852035] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 758.744712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.758565] LNet: 46353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.767903] LNetError: 46353:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.782963] LNet: Removed LNI 192.168.204.5@tcp [ 758.809617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.874513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 758.924512] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 766.163412] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 14:11:03 (1782756663) [ 766.479861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.495212] LNet: 46951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.504744] LNetError: 46951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 766.537564] LNet: Removed LNI 192.168.204.5@tcp [ 766.559287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.671697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 773.268987] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 14:11:10 (1782756670) [ 773.416992] LNet: 47528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.213588] Key type .llcrypt unregistered [ 774.223351] Key type ._llcrypt unregistered [ 781.499603] Key type ._llcrypt registered [ 781.507588] Key type .llcrypt registered [ 781.549916] LNet: 47760:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 781.549916] mask usage: [+|-] ... [ 781.719485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 781.740095] alg: No test for adler32 (adler32-zlib) [ 782.660222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.731509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 782.778717] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 783.682406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.704513] LNet: 47932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.710896] LNetError: 47932:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.736182] LNet: Removed LNI 192.168.204.5@tcp [ 783.757834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.817572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 783.845811] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 791.578337] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 14:11:28 (1782756688) [ 791.741309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 791.772486] LNet: 48522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.782809] LNetError: 48522:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.830814] LNet: Removed LNI 192.168.204.5@tcp [ 791.868722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.960236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 798.545731] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 14:11:35 (1782756695) [ 800.296440] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 802.403152] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 14:11:39 (1782756699) [ 804.146905] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 806.001417] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 14:11:42 (1782756702) [ 807.797318] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 809.837257] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 14:11:46 (1782756706) [ 811.646575] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 813.539620] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 14:11:50 (1782756710) [ 815.569560] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 817.946443] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 14:11:54 (1782756714) [ 818.076908] LNet: 50572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.998410] Key type .llcrypt unregistered [ 819.002134] Key type ._llcrypt unregistered [ 825.802225] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 825.964158] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 826.000776] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 826.005658] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 828.650463] Key type ._llcrypt registered [ 828.653297] Key type .llcrypt registered [ 828.676372] LNet: 50856:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 828.676372] mask usage: [+|-] ... [ 828.890918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 828.911180] alg: No test for adler32 (adler32-zlib) [ 829.822364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.915120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 829.963407] LNet: Added LNI 192.168.204.5@tcp [8/512/0/180] [ 829.976554] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 829.979923] LNet: 50894:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 830.111369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.135402] LNet: 50913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.146769] LNetError: 50913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 830.153629] LNet: Removed LNI 10.1.2.3@tcp [ 830.158154] LNetError: 50913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.191807] LNet: Removed LNI 192.168.204.5@tcp [ 830.223682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.311908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 830.343423] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 830.455423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.493255] LNet: 50938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.504923] LNetError: 50938:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.528189] LNet: Removed LNI 192.168.204.5@tcp [ 830.554810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.645494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 830.684701] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 830.689837] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 830.693650] LNet: 50946:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 830.798637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.819271] LNet: 50964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.824711] LNetError: 50964:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 830.830563] LNet: Removed LNI 10.1.2.3@tcp [ 830.832876] LNetError: 50964:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.854167] LNet: Removed LNI 192.168.204.5@tcp [ 830.880159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.030021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 831.074526] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 831.079925] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 831.082060] LNet: 50972:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 831.259710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 831.285165] LNet: 50998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.294136] LNetError: 50998:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 831.303180] LNet: Removed LNI 10.1.2.3@tcp [ 831.305531] LNetError: 50998:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 831.317740] LNet: Removed LNI 192.168.204.5@tcp [ 831.339906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.424933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 831.454636] LNet: Added LNI 192.168.204.5@tcp [8/256/0/182] [ 831.463786] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 831.581509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 831.606378] LNet: 51026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.625838] LNetError: 51026:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 831.635993] LNet: Removed LNI 10.1.2.3@tcp [ 831.637905] LNetError: 51026:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 831.655750] LNet: Removed LNI 192.168.204.5@tcp [ 831.677439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.752448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 831.797225] LNet: Added LNI 192.168.204.5@tcp [8/256/0/182] [ 831.804637] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 831.809934] LNet: 51034:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 831.936810] LNet: Removed LNI 10.1.2.3@tcp [ 831.968514] LNet: Removed LNI 192.168.204.5@tcp [ 832.571570] Key type .llcrypt unregistered [ 832.575520] Key type ._llcrypt unregistered [ 843.486261] Key type ._llcrypt registered [ 843.489612] Key type .llcrypt registered [ 845.049853] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 14:12:21 (1782756741) [ 845.309283] LNet: 51858:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 845.309283] mask usage: [+|-] ... [ 845.473887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 845.487561] alg: No test for adler32 (adler32-zlib) [ 846.430663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.484513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 846.511593] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 847.257376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.5 [ 847.274307] LNet: 52020:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 847.282076] LNet: Removed LNI 192.168.204.5@tcp [ 853.296885] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 14:12:30 (1782756750) [ 853.538138] LNet: 52590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.557952] LNetError: 52590:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.588345] LNet: Removed LNI 192.168.204.5@tcp [ 854.248348] Key type .llcrypt unregistered [ 854.253449] Key type ._llcrypt unregistered [ 861.030383] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 861.280429] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 864.053868] Key type ._llcrypt registered [ 864.057836] Key type .llcrypt registered [ 864.125322] LNet: 52873:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 864.125322] mask usage: [+|-] ... [ 864.319275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 864.341268] alg: No test for adler32 (adler32-zlib) [ 865.691727] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 872.591814] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 14:12:49 (1782756769) [ 872.684841] LNet: 53496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.689698] LNetError: 53496:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 872.716301] LNet: Removed LNI 10.1.2.3@tcp [ 873.617129] Key type .llcrypt unregistered [ 873.618672] Key type ._llcrypt unregistered [ 880.933669] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 881.349226] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 881.353839] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 881.364958] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 885.012710] Key type ._llcrypt registered [ 885.017869] Key type .llcrypt registered [ 885.068837] LNet: 53778:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 885.068837] mask usage: [+|-] ... [ 885.315422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 885.359389] alg: No test for adler32 (adler32-zlib) [ 886.479948] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 897.982239] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 14:13:12 (1782756792) [ 901.047669] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 903.365506] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 14:13:19 (1782756799) [ 903.520442] LNet: 54684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.543894] LNetError: 54684:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 903.609660] LNet: Removed LNI 10.1.2.3@tcp [ 904.347454] Key type .llcrypt unregistered [ 904.349253] Key type ._llcrypt unregistered [ 910.300184] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 910.527961] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 910.533340] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 910.543239] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 913.562852] Key type ._llcrypt registered [ 913.566458] Key type .llcrypt registered [ 913.629130] LNet: 54967:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 913.629130] mask usage: [+|-] ... [ 913.811891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 913.830329] alg: No test for adler32 (adler32-zlib) [ 914.730039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.794641] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 924.633745] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 14:13:41 (1782756821) [ 925.170551] LNet: 55923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.174986] LNetError: 55923:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 925.188104] LNet: Removed LNI 10.1.2.3@tcp [ 925.759129] Key type .llcrypt unregistered [ 925.762696] Key type ._llcrypt unregistered [ 933.587191] Key type ._llcrypt registered [ 933.591225] Key type .llcrypt registered [ 933.713086] LNet: 56147:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 933.713086] mask usage: [+|-] ... [ 934.079168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 934.149866] alg: No test for adler32 (adler32-zlib) [ 935.123372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 935.178136] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 947.571185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.105@tcp [ 948.480382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_interrupt -r 1 [ 948.538197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_interrupt -r 1 [ 948.609821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 948.639524] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 948.952699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 948.991244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 949.048918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 949.114691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 949.166934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 949.248975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 949.315493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 949.399926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 949.449697] LNet: 56363:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 949.472134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 949.515750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 949.545395] LNet: 1 local NIs in recovery (showing 1): 192.168.204.5@tcp [ 949.563982] LNet: There was an unexpected network error while writing to 192.168.204.5: rc = -22 [ 949.588904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 949.679957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 949.734967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 949.788594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 949.844608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 950.444084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_dropped -r 1 [ 950.468330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_dropped -r 1 [ 950.492166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 950.707877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 950.746451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 950.796682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 950.828079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 950.852631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 950.879156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 950.905165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 950.934575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 950.952948] LNet: 56472:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 950.969032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 951.001668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 951.031660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 951.059092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 951.101629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 951.135940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 951.183742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 951.735906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_aborted -r 1 [ 951.789737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_aborted -r 1 [ 951.863873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 951.928647] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 951.941064] LNet: Skipped 1 previous similar message [ 952.398308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 952.529235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 952.593178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 952.660583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 952.721606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 952.775909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 952.843365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 952.899654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 952.928411] LNet: 56581:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 952.956548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 952.994478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 953.048629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 953.100877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 953.156619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 953.206153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 953.254275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 953.897236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_no_route -r 1 [ 953.976536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_no_route -r 1 [ 954.022264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 954.063553] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 954.424094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 954.474967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 954.534665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 954.606925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 954.671892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 954.732201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 954.763955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 954.800684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 954.836068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 954.871535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 954.908516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 954.945285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 954.973603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 955.002400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 955.051110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 955.610379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_timeout -r 1 [ 955.668552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_timeout -r 1 [ 955.749254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 956.216631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 956.262129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 956.358831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 956.415422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 956.478992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 956.583849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 956.668574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 956.772812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 956.809509] LNet: 56800:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 956.820177] LNet: 56800:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 956.865572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 956.927052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 956.992360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 957.065254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 957.128220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 957.199360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 957.266664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 957.981034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_error -r 1 [ 958.057864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_error -r 1 [ 958.093640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 958.113504] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 958.119990] LNet: Skipped 1 previous similar message [ 958.308194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 958.327411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 958.353928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 958.385142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 958.409057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 958.436662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 958.470485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 958.499430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 958.524164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 958.550551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 958.572158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 958.598947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 958.628613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 958.653606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 958.682522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 958.877835] LNet: 56951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.882280] LNetError: 56951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.915054] LNet: Removed LNI 192.168.204.5@tcp [ 960.320454] Key type .llcrypt unregistered [ 960.325436] Key type ._llcrypt unregistered [ 978.181890] Key type ._llcrypt registered [ 978.183294] Key type .llcrypt registered [ 979.383802] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 14:14:36 (1782756876) [ 981.121749] Key type .llcrypt unregistered [ 981.123603] Key type ._llcrypt unregistered [ 989.282384] Key type ._llcrypt registered [ 989.285834] Key type .llcrypt registered [ 989.317347] LNet: 57990:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 989.317347] mask usage: [+|-] ... [ 989.515063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 989.539509] alg: No test for adler32 (adler32-zlib) [ 990.559460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 990.596352] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1000.227281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.105@tcp [ 1013.643620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.666882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1013.680612] LNet: Added LNI 192.168.204.5@tcp1 [8/256/0/180] [ 1015.233679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_interrupt -r 1 [ 1015.292213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp1 -m GET -e local_interrupt -r 1 [ 1015.352060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_interrupt -r 1 [ 1015.400530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -m GET -e local_interrupt -r 1 [ 1015.458600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp -m GET -e local_interrupt -r 1 [ 1015.503202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp1 -m GET -e local_interrupt -r 1 [ 1015.545855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -m GET -e local_interrupt -r 1 [ 1015.584145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -m GET -e local_interrupt -r 1 [ 1015.622965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 1015.641689] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1015.647729] LNet: 1 local NIs in recovery (showing 1): 192.168.204.5@tcp [ 1015.861396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1015.928611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1015.971676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1016.005411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1016.041195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1016.065991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1016.100409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1016.126473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1016.141989] LNet: 58420:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1016.159459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1016.218550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1016.264727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1016.322738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1016.356788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1016.416508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1016.489445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1017.229739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_dropped -r 1 [ 1017.366698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp1 -m GET -e local_dropped -r 1 [ 1017.457592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_dropped -r 1 [ 1017.521595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -m GET -e local_dropped -r 1 [ 1017.607255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp -m GET -e local_dropped -r 1 [ 1017.674230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp1 -m GET -e local_dropped -r 1 [ 1017.734595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -m GET -e local_dropped -r 1 [ 1017.818611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -m GET -e local_dropped -r 1 [ 1017.882497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 1017.935295] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1017.943247] LNet: Skipped 2 previous similar messages [ 1018.237776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1018.261292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1018.285383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1018.312316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1018.341601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1018.361488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1018.385283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1018.423786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1018.453824] LNet: 58541:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1018.494522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1018.514297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1018.536359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1018.580478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1018.630622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1018.696587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1018.756368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1019.600917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_aborted -r 1 [ 1019.660867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp1 -m GET -e local_aborted -r 1 [ 1019.720038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_aborted -r 1 [ 1019.776206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -m GET -e local_aborted -r 1 [ 1019.814769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp -m GET -e local_aborted -r 1 [ 1019.858418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp1 -m GET -e local_aborted -r 1 [ 1019.930099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -m GET -e local_aborted -r 1 [ 1019.973786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -m GET -e local_aborted -r 1 [ 1020.022413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 1020.048113] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1020.055450] LNet: Skipped 2 previous similar messages [ 1020.274521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1020.317786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1020.370231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1020.402350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1020.429855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1020.472838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1020.539924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1020.591725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1020.635881] LNet: 58662:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1020.660938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1020.710421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1020.770444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1020.815731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1020.863993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1020.905822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1020.961929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1021.710723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_no_route -r 1 [ 1021.767425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp1 -m GET -e local_no_route -r 1 [ 1021.824187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_no_route -r 1 [ 1021.872693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -m GET -e local_no_route -r 1 [ 1021.970701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp -m GET -e local_no_route -r 1 [ 1022.036165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp1 -m GET -e local_no_route -r 1 [ 1022.109763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -m GET -e local_no_route -r 1 [ 1022.174563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -m GET -e local_no_route -r 1 [ 1022.242448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 1022.283206] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1022.296590] LNet: Skipped 2 previous similar messages [ 1022.722844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1022.771724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1022.830345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1022.894454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1022.964348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1023.041128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1023.145693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1023.234050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1023.249168] LNet: 58784:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1023.279804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1023.332812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1023.406550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1023.465703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1023.509757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1023.581264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1023.641534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1024.340124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_timeout -r 1 [ 1024.392691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp1 -m GET -e local_timeout -r 1 [ 1024.452560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_timeout -r 1 [ 1024.516430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -m GET -e local_timeout -r 1 [ 1024.565843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp -m GET -e local_timeout -r 1 [ 1024.617679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp1 -m GET -e local_timeout -r 1 [ 1024.660241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -m GET -e local_timeout -r 1 [ 1024.721701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -m GET -e local_timeout -r 1 [ 1024.801416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 1025.092925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1025.120784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1025.153462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1025.182630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1025.227352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1025.280826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1025.310829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1025.338970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1025.372170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1025.413225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1025.449789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1025.504809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1025.586813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1025.628181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1025.667832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1026.595389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_error -r 1 [ 1026.622473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp1 -m GET -e local_error -r 1 [ 1026.665682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_error -r 1 [ 1026.708680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -m GET -e local_error -r 1 [ 1026.766381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp -m GET -e local_error -r 1 [ 1026.868613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp1 -m GET -e local_error -r 1 [ 1026.946309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -m GET -e local_error -r 1 [ 1027.000674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -m GET -e local_error -r 1 [ 1027.063356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 1027.085732] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1027.103910] LNet: Skipped 7 previous similar messages [ 1027.502308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1027.569318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1027.619900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1027.674797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1027.725986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1027.835692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1027.923495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1028.047907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1028.085714] LNet: 59027:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1028.098292] LNet: 59027:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1028.147794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1028.195053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1028.242180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1028.296204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1028.331351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1028.355494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1028.386522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1034.894615] LNet: 59103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.899295] LNetError: 59103:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1034.909511] LNet: Removed LNI 192.168.204.5@tcp [ 1035.944282] LNet: Removed LNI 192.168.204.5@tcp1 [ 1036.341615] Key type .llcrypt unregistered [ 1036.344578] Key type ._llcrypt unregistered [ 1054.479955] Key type ._llcrypt registered [ 1054.485516] Key type .llcrypt registered [ 1055.939561] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 14:15:52 (1782756952) [ 1056.637826] Key type .llcrypt unregistered [ 1056.643733] Key type ._llcrypt unregistered [ 1062.816885] Key type ._llcrypt registered [ 1062.819064] Key type .llcrypt registered [ 1062.879483] LNet: 60138:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1062.879483] mask usage: [+|-] ... [ 1063.064338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1063.081376] alg: No test for adler32 (adler32-zlib) [ 1064.006448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1064.026527] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1071.536225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.105@tcp [ 1072.150314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e remote_dropped -r 1 [ 1072.197543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e remote_dropped -r 1 [ 1072.254376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 1072.268524] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1072.536728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1072.577271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1072.623277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1072.694395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1072.751774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1072.795476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1072.832571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1072.885727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1072.917465] LNet: 60352:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1072.967528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1073.016080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1073.075375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1073.136585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1073.187301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1073.229126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1073.281830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1073.743683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e remote_error -r 1 [ 1073.767877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e remote_error -r 1 [ 1073.815738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 1073.846374] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1074.127767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1074.173374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1074.224151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1074.302971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1074.349660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1074.380278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1074.437516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1074.493526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1074.514308] LNet: 60461:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1074.542760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1074.581265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1074.602988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1074.645781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1074.695245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1074.729780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1074.778453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1075.215733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e remote_timeout -r 1 [ 1075.257352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e remote_timeout -r 1 [ 1075.293622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 1075.314413] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1075.548354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1075.581895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1075.614686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1075.659910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1075.704308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1075.732832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1075.759318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1075.790776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1075.809326] LNet: 60571:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1075.828133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1075.858264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1075.890786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1075.920493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1075.949648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1075.975473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1076.015522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1076.334286] LNet: 60612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.345573] LNetError: 60612:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1076.368895] LNet: Removed LNI 192.168.204.5@tcp [ 1076.770560] Key type .llcrypt unregistered [ 1076.773854] Key type ._llcrypt unregistered [ 1092.269377] Key type ._llcrypt registered [ 1092.273304] Key type .llcrypt registered [ 1093.348169] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 14:16:30 (1782756990) [ 1094.058819] Key type .llcrypt unregistered [ 1094.065764] Key type ._llcrypt unregistered [ 1100.615401] Key type ._llcrypt registered [ 1100.620355] Key type .llcrypt registered [ 1100.674585] LNet: 61650:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1100.674585] mask usage: [+|-] ... [ 1100.921407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1100.966765] alg: No test for adler32 (adler32-zlib) [ 1101.914387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1101.944595] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1109.406706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.105@tcp [ 1118.567204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1118.628960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1118.654451] LNet: Added LNI 192.168.204.5@tcp1 [8/256/0/180] [ 1120.465452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e remote_dropped -r 1 [ 1120.519531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp1 -m GET -e remote_dropped -r 1 [ 1120.570102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e remote_dropped -r 1 [ 1120.639486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -m GET -e remote_dropped -r 1 [ 1120.704932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp -m GET -e remote_dropped -r 1 [ 1120.754576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp1 -m GET -e remote_dropped -r 1 [ 1120.807328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -m GET -e remote_dropped -r 1 [ 1120.846456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -m GET -e remote_dropped -r 1 [ 1120.891729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 1120.939920] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1120.952921] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.105@tcp [ 1121.195740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.237918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1121.283096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1121.339238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1121.386398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1121.430393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1121.471849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1121.506221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1121.537194] LNet: 62079:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1121.557342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1121.582267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1121.619222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1121.658341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1121.687841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1121.714689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1121.744517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1121.789530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.287061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e remote_error -r 1 [ 1122.318080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp1 -m GET -e remote_error -r 1 [ 1122.345934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e remote_error -r 1 [ 1122.375992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -m GET -e remote_error -r 1 [ 1122.416695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp -m GET -e remote_error -r 1 [ 1122.447203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp1 -m GET -e remote_error -r 1 [ 1122.485892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -m GET -e remote_error -r 1 [ 1122.528958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -m GET -e remote_error -r 1 [ 1122.569620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 1122.598856] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1122.604502] LNet: Skipped 2 previous similar messages [ 1122.781511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1122.806569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1122.831982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.854208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1122.876921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1122.922668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.949913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1123.012699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1123.028952] LNet: 62202:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1123.045079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1123.095886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1123.128755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1123.160185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1123.195285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.233350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1123.262939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1123.302280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.863100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e remote_timeout -r 1 [ 1123.902165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp1 -m GET -e remote_timeout -r 1 [ 1123.934984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e remote_timeout -r 1 [ 1123.972908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -m GET -e remote_timeout -r 1 [ 1124.003226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp -m GET -e remote_timeout -r 1 [ 1124.037318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp1 -m GET -e remote_timeout -r 1 [ 1124.077626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -m GET -e remote_timeout -r 1 [ 1124.142953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -m GET -e remote_timeout -r 1 [ 1124.201256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.105@tcp [ 1124.227252] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1124.232158] LNet: Skipped 1 previous similar message [ 1124.423785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.454816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1124.486642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.528943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.577535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.608813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.630832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.655440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.677547] LNet: 62326:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.698080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.726756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1124.757518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.781427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.818465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.852511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.881547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1124.932460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1129.715235] LNet: 62403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.726310] LNetError: 62403:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.730699] LNet: Removed LNI 192.168.204.5@tcp [ 1129.746326] LNet: Removed LNI 192.168.204.5@tcp1 [ 1130.219510] Key type .llcrypt unregistered [ 1130.224715] Key type ._llcrypt unregistered [ 1147.668781] Key type ._llcrypt registered [ 1147.672443] Key type .llcrypt registered [ 1149.248975] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 14:17:25 (1782757045) [ 1150.769382] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1152.374449] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 14:17:29 (1782757049) [ 1153.411565] Key type .llcrypt unregistered [ 1153.416627] Key type ._llcrypt unregistered [ 1160.258573] Key type ._llcrypt registered [ 1160.260548] Key type .llcrypt registered [ 1160.310880] LNet: 63732:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1160.310880] mask usage: [+|-] ... [ 1160.553107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1160.579813] alg: No test for adler32 (adler32-zlib) [ 1161.572906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1161.634210] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1170.630418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.105@tcp [ 1171.453571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e network_timeout -r 1 [ 1171.509285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e network_timeout -r 1 [ 1171.577951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.105@tcp [ 1171.605771] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1171.872285] LNet: 1 local NIs in recovery (showing 1): 192.168.204.5@tcp [ 1171.891338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.932739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1171.991911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1172.036627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1172.077425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1172.112718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1172.152748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1172.217623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1172.251968] LNet: 63946:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1172.281712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1172.316701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1172.366855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.406755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1172.450345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.479563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.794387] LNet: 63986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.801967] LNetError: 63986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.818804] LNet: Removed LNI 192.168.204.5@tcp [ 1173.236358] Key type .llcrypt unregistered [ 1173.241731] Key type ._llcrypt unregistered [ 1193.562389] Key type ._llcrypt registered [ 1193.564676] Key type .llcrypt registered [ 1193.609868] LNet: 64448:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1193.609868] mask usage: [+|-] ... [ 1193.719229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1193.732604] alg: No test for adler32 (adler32-zlib) [ 1194.639504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1194.657791] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1202.551642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.105@tcp [ 1211.767578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1211.829473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1211.880421] LNet: Added LNI 192.168.204.5@tcp1 [8/256/0/180] [ 1213.271963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e network_timeout -r 1 [ 1213.330866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp1 -m GET -e network_timeout -r 1 [ 1213.370315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e network_timeout -r 1 [ 1213.424347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -m GET -e network_timeout -r 1 [ 1213.481170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp -m GET -e network_timeout -r 1 [ 1213.522733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp1 -m GET -e network_timeout -r 1 [ 1213.560353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -m GET -e network_timeout -r 1 [ 1213.618434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -m GET -e network_timeout -r 1 [ 1213.676695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.105@tcp [ 1213.707771] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1214.019900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1214.063526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1214.113469] LNet: 1 local NIs in recovery (showing 1): 192.168.204.5@tcp [ 1214.128205] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.105@tcp [ 1214.150225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1214.199141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1214.228491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1214.261675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1214.300429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1214.329966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1214.345185] LNet: 64877:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1214.366476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1214.407901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1214.448316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1214.483634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1214.511403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1214.540113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1219.884153] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1219.898091] LNet: Skipped 2 previous similar messages [ 1220.215268] LNet: 64950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1220.221506] LNetError: 64950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.281883] LNet: Removed LNI 192.168.204.5@tcp [ 1221.284338] LNetError: 64950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.308881] LNet: Removed LNI 192.168.204.5@tcp1 [ 1221.760533] Key type .llcrypt unregistered [ 1221.763807] Key type ._llcrypt unregistered [ 1240.213892] Key type ._llcrypt registered [ 1240.216665] Key type .llcrypt registered [ 1241.786843] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 14:18:58 (1782757138) [ 1242.107970] LNet: 65751:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1242.107970] mask usage: [+|-] ... [ 1242.242804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1242.275422] alg: No test for adler32 (adler32-zlib) [ 1243.178500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1243.230396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1243.274151] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1244.128708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1244.171621] LNet: Added LNI 192.168.204.5@tcp1 [8/256/0/180] [ 1245.081304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 1245.190198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1245.277166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1245.336610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1245.399925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1245.463312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1245.567077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.5@tcp [ 1245.624250] LNetError: 66050:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.5@tcp to recovery [ 1246.306507] LNet: 1 local NIs in recovery (showing 1): 192.168.204.5@tcp [ 1246.307482] LNet: There was an unexpected network error while writing to 192.168.204.5: rc = -22 [ 1248.353085] LNet: There was an unexpected network error while writing to 192.168.204.5: rc = -22 [ 1249.473863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1252.451193] LNet: There was an unexpected network error while writing to 192.168.204.5: rc = -22 [ 1253.298898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1253.379350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1253.447791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1253.469912] LNet: 66157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.477287] LNetError: 66157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.484553] LNet: Removed LNI 192.168.204.5@tcp [ 1253.499562] LNet: Removed LNI 192.168.204.5@tcp1 [ 1253.550519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1253.638544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1253.659463] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1254.505412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1254.533745] LNet: Added LNI 192.168.204.5@tcp1 [8/256/0/180] [ 1255.434643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 1255.538208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1255.643967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1255.715553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1255.760198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1255.820658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1255.854954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1255.907400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.5@tcp [ 1255.931621] LNetError: 66430:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.5@tcp to recovery [ 1256.672356] LNet: There was an unexpected network error while writing to 192.168.204.5: rc = -22 [ 1259.751713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1262.819061] LNet: There was an unexpected network error while writing to 192.168.204.5: rc = -22 [ 1262.834121] LNet: Skipped 1 previous similar message [ 1267.238788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1267.334129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1267.425149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1277.873422] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 14:19:34 (1782757174) [ 1278.227945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1278.292330] LNet: 67137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.318763] LNetError: 67137:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1278.338209] LNetError: 67137:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1278.359101] LNet: Removed LNI 192.168.204.5@tcp [ 1278.382380] LNet: Removed LNI 192.168.204.5@tcp1 [ 1278.448197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.636908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1278.733398] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1280.621329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1280.689824] LNet: Added LNI 192.168.204.5@tcp1 [8/256/0/180] [ 1281.892668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 1281.996238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1282.023424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1282.053623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1282.078254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1282.110768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1282.162270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.5@tcp --health 0 [ 1282.311634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1282.656265] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.5@tcp [ 1283.680450] LNet: There was an unexpected network error while writing to 192.168.204.5: rc = -22 [ 1283.686552] LNet: Skipped 1 previous similar message [ 1298.134759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1298.330402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1298.387890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1298.422310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.5@tcp --health 500 [ 1298.561203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1305.261123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1305.360494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1305.409429] LNet: 67629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1305.431865] LNetError: 67629:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1305.439777] LNetError: 67629:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1305.447578] LNet: Removed LNI 192.168.204.5@tcp [ 1305.464854] LNet: Removed LNI 192.168.204.5@tcp1 [ 1305.501572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1305.561358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1305.591918] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1306.296278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1306.319502] LNet: Added LNI 192.168.204.5@tcp1 [8/256/0/180] [ 1307.273552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 1307.310105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1307.357435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1307.382622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1307.409798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1307.436480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1307.463926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1307.494745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.5@tcp --health 0 [ 1308.640668] LNet: There was an unexpected network error while writing to 192.168.204.5: rc = -22 [ 1308.647262] LNet: Skipped 3 previous similar messages [ 1308.854251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1319.473734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1319.535774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1319.580243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1332.458140] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 14:20:27 (1782757227) [ 1333.249335] LNet: 68675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1333.288504] LNetError: 68675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1333.297512] LNetError: 68675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1333.305490] LNet: Removed LNI 192.168.204.5@tcp [ 1333.323466] LNet: Removed LNI 192.168.204.5@tcp1 [ 1334.430918] Key type .llcrypt unregistered [ 1334.433923] Key type ._llcrypt unregistered [ 1343.364698] Key type ._llcrypt registered [ 1343.369396] Key type .llcrypt registered [ 1343.462474] LNet: 68899:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1343.462474] mask usage: [+|-] ... [ 1343.791640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1343.823561] alg: No test for adler32 (adler32-zlib) [ 1344.996156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1345.060840] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1355.037445] Lustre: DEBUG MARKER: Initial discovery [ 1355.151734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.105@tcp [ 1357.794152] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1357.870247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1357.925493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.105@tcp [ 1357.962591] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1358.003106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1358.074993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.105@tcp -r 1 -m GET -l 3 [ 1358.158143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.105@tcp -r 1 -m GET -e local_error [ 1358.211584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.105@tcp -r 1 -m PUT -l 6 [ 1358.369166] LNet: 1 local NIs in recovery (showing 1): 192.168.204.5@tcp [ 1359.827798] Lustre: DEBUG MARKER: Force oleg405-client.virtnet to discover 192.168.204.105@tcp (in background) [ 1359.914353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.105@tcp [ 1361.177947] Lustre: DEBUG MARKER: Force oleg405-server to discover 192.168.204.5@tcp [ 1363.050080] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1364.203909] Lustre: DEBUG MARKER: Wait for 69430 [ 1365.551723] Lustre: DEBUG MARKER: Finished wait on 69430 [ 1365.844597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1365.885529] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1365.887883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1366.267582] LNet: 69889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1366.283401] LNetError: 69889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1366.315954] LNet: Removed LNI 192.168.204.5@tcp [ 1366.929608] Key type .llcrypt unregistered [ 1366.935539] Key type ._llcrypt unregistered [ 1385.990786] Key type ._llcrypt registered [ 1386.002328] Key type .llcrypt registered [ 1388.122113] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 14:21:24 (1782757284) [ 1388.621386] Key type .llcrypt unregistered [ 1388.625197] Key type ._llcrypt unregistered [ 1397.377535] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1397.495789] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1397.507186] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1397.518929] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1399.243682] Key type ._llcrypt registered [ 1399.245584] Key type .llcrypt registered [ 1399.318778] LNet: 70925:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1399.318778] mask usage: [+|-] ... [ 1399.756372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1399.802726] alg: No test for adler32 (adler32-zlib) [ 1400.703755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1400.770772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1400.824817] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1402.010765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1402.050432] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1411.980822] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 14:21:48 (1782757308) [ 1412.507475] LNet: 71959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.514359] LNetError: 71959:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1412.527664] LNet: Removed LNI 10.1.2.3@tcp [ 1412.540646] LNet: Removed LNI 192.168.204.5@tcp [ 1413.218410] Key type .llcrypt unregistered [ 1413.221565] Key type ._llcrypt unregistered [ 1421.019075] Key type ._llcrypt registered [ 1421.027120] Key type .llcrypt registered [ 1421.109211] LNet: 72208:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1421.109211] mask usage: [+|-] ... [ 1421.321449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1421.345571] alg: No test for adler32 (adler32-zlib) [ 1422.303347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1422.368308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1422.387758] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1423.151939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1423.189261] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1424.422888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.5@tcp [ 1425.504160] LNetError: Unexpected error -22 connecting to 192.168.204.5@tcp at host 192.168.204.5:988 [ 1432.890728] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 14:22:09 (1782757329) [ 1433.434438] LNet: 73092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1433.454390] LNetError: 73092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1433.478913] LNet: Removed LNI 10.1.2.3@tcp [ 1433.498261] LNet: Removed LNI 192.168.204.5@tcp [ 1434.277498] Key type .llcrypt unregistered [ 1434.280140] Key type ._llcrypt unregistered [ 1440.552284] Key type ._llcrypt registered [ 1440.554592] Key type .llcrypt registered [ 1440.674985] LNet: 73322:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1440.674985] mask usage: [+|-] ... [ 1440.877886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1440.886625] alg: No test for adler32 (adler32-zlib) [ 1441.845377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1441.888945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1441.905076] LNet: Added LNI 192.168.204.5@tcp1 [8/256/0/180] [ 1442.487324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1442.520798] LNet: Added LNI 192.168.204.5@tcp2 [8/256/0/180] [ 1443.650910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.5@tcp1 --nid 192.168.204.5@tcp2 [ 1447.706747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.5@tcp1 192.168.204.5@tcp2 [ 1447.724473] LNetError: 73926:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.5@tcp1 to 192.168.204.5@tcp2 - Route table may be misconfigured [ 1447.744179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.5@tcp2 192.168.204.5@tcp1 [ 1454.718387] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 14:22:31 (1782757351) [ 1454.918806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1454.956450] LNet: 74500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.963211] LNetError: 74500:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1454.972455] LNet: Removed LNI 192.168.204.5@tcp1 [ 1456.051675] LNet: Removed LNI 192.168.204.5@tcp2 [ 1456.107525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1456.180555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1456.212826] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1457.082321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1457.114996] LNet: Added LNI 192.168.204.5@tcp1 [8/256/0/180] [ 1457.971796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 1458.087565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.5@tcp 192.168.204.5@tcp [ 1458.195193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.5@tcp1 192.168.204.5@tcp1 [ 1458.258398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.5@tcp -d 192.168.204.5@tcp -e network_timeout [ 1458.307219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -e network_timeout [ 1458.345448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -e network_timeout [ 1458.381839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -e network_timeout [ 1458.420857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 1458.445727] LNet: There was an unexpected network error while writing to 192.168.204.5: rc = -22 [ 1458.464667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1458.539923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1458.635840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1459.237170] LNet: 1 local NIs in recovery (showing 1): 192.168.204.5@tcp [ 1465.663558] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 14:22:42 (1782757362) [ 1466.042815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1466.089241] LNet: 75362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.099518] LNetError: 75362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.112753] LNetError: 75362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1466.143560] LNet: Removed LNI 192.168.204.5@tcp [ 1467.175482] LNet: Removed LNI 192.168.204.5@tcp1 [ 1467.221333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1467.367909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1467.719137] LNet: 75399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1468.093365] Key type .llcrypt unregistered [ 1468.095065] Key type ._llcrypt unregistered [ 1481.052587] Key type ._llcrypt registered [ 1481.054494] Key type .llcrypt registered [ 1482.747250] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 14:22:59 (1782757379) [ 1482.955315] LNet: 76166:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1482.955315] mask usage: [+|-] ... [ 1483.178250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1483.204536] alg: No test for adler32 (adler32-zlib) [ 1484.231865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1485.856599] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1488.020298] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 14:23:04 (1782757384) [ 1488.263595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1488.298263] LNet: 76497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.333985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1488.432781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1488.456349] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1489.604256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1489.631458] LNet: Added LNI 192.168.204.5@tcp1 [8/256/0/180] [ 1491.100562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 1491.156182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp1 [ 1491.223228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp1 [ 1498.928140] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 14:23:16 (1782757396) [ 1500.906288] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg405-server [ 1503.199341] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 14:23:20 (1782757400) [ 1505.676789] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg405-server [ 1507.772491] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 14:23:24 (1782757404) [ 1509.829771] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg405-server [ 1511.696935] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 14:23:28 (1782757408) [ 1513.928279] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg405-server [ 1515.778569] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 14:23:32 (1782757412) [ 1517.966539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg405-server [ 1520.523303] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 14:23:36 (1782757416) [ 1522.810136] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg405-server [ 1524.767657] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 14:23:41 (1782757421) [ 1526.677895] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg405-server [ 1528.642849] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 14:23:45 (1782757425) [ 1530.554108] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg405-server [ 1532.376734] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 14:23:49 (1782757429) [ 1534.252737] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg405-server [ 1535.824387] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 14:23:52 (1782757432) [ 1535.922898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.958752] LNet: 80369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.963703] LNetError: 80369:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.970392] LNet: Removed LNI 192.168.204.5@tcp [ 1537.011271] LNet: Removed LNI 192.168.204.5@tcp1 [ 1537.042312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.112644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1537.162972] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1537.864596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1537.935454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 1538.174705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1538.238163] LNet: 80522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1538.246901] LNetError: 80522:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1538.255328] LNetError: 80522:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1538.279550] LNet: Removed LNI 192.168.204.5@tcp [ 1538.322749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.423552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1538.488507] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1539.385392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 1539.590155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1539.638597] LNetError: 80668:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1539.667842] LNet: Removed LNI 192.168.204.5@tcp [ 1539.711942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.770993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1539.826462] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1540.824529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1540.940383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 1541.298444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1541.333763] LNet: 80815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.343719] LNet: 80815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1542.378447] LNet: Removed LNI 192.168.204.5@tcp [ 1542.430876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.557593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1542.627480] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1543.565930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 1543.880081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1543.932152] LNetError: 80960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.947083] LNetError: 80960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1543.979049] LNet: Removed LNI 192.168.204.5@tcp [ 1544.046549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1544.177375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1544.231255] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1545.450216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1545.566732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 1545.884679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1545.921540] LNet: 81107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.932108] LNet: 81107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1546.982977] LNet: Removed LNI 192.168.204.5@tcp [ 1547.036396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.101348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1547.118528] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1548.230588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 1548.584517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.644066] LNetError: 81253:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1548.650736] LNetError: 81253:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1548.670861] LNet: Removed LNI 192.168.204.5@tcp [ 1548.725402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1548.814443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1548.867492] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1549.607996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1549.693358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 1549.997406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.086307] LNet: Removed LNI 192.168.204.5@tcp [ 1551.122553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.209827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1551.239056] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1552.214973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 1552.591590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1553.639946] LNet: Removed LNI 192.168.204.5@tcp [ 1553.676939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1553.750659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1553.775695] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1554.497277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1554.613881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 1554.863920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1554.890684] LNet: 81693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1554.896122] LNet: 81693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1555.948076] LNet: Removed LNI 192.168.204.5@tcp [ 1555.984958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1556.043913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1556.077969] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1556.835521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 1557.257969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.312854] LNetError: 81838:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.320860] LNetError: 81838:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1558.381273] LNet: Removed LNI 192.168.204.5@tcp [ 1558.434819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1558.521312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1558.557098] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1559.435544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1559.544502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 1560.998422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.090808] LNet: Removed LNI 192.168.204.5@tcp [ 1562.116881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.240893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1562.299990] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1563.595063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 1563.950150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.037770] LNet: Removed LNI 192.168.204.5@tcp [ 1565.089570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.253593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1565.317050] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1566.455470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1566.569265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1566.679282] LNet: Removed LNI 192.168.204.5@tcp [ 1566.729527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.813437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1566.850344] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1567.968457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1575.793933] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 14:24:32 (1782757472) [ 1575.953803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.004824] LNet: 82971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.010533] LNet: 82971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1576.020804] LNetError: 82971:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1576.033503] LNetError: 82971:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1576.062517] LNet: Removed LNI 192.168.204.5@tcp [ 1576.112899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.177518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1576.206547] LNet: Added LNI 192.168.204.5@tcp231 [8/256/0/180] [ 1577.384248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.460043] LNet: Removed LNI 192.168.204.5@tcp231 [ 1577.478280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.555813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1577.598099] LNet: Added LNI 192.168.204.5@tcp231 [8/256/0/180] [ 1577.610059] LNet: 83123:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1578.256983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1578.280661] LNet: Removed LNI 192.168.204.5@tcp231 [ 1578.295371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1578.326926] LNet: Added LNI 192.168.204.5@tcp231 [8/256/0/180] [ 1579.836076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.883107] LNet: Removed LNI 192.168.204.5@tcp231 [ 1579.931023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.009796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1580.062212] LNet: Added LNI 192.168.204.5@tcp231 [8/256/0/0] [ 1580.069151] LNet: 83276:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1580.654582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1580.697426] LNet: Removed LNI 192.168.204.5@tcp231 [ 1580.724778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1580.763591] LNet: Added LNI 192.168.204.5@tcp231 [8/256/0/0] [ 1581.926933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.967380] LNet: Removed LNI 192.168.204.5@tcp231 [ 1581.990737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.087492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1582.117980] LNet: Added LNI 192.168.204.5@tcp231 [8/256/0/60] [ 1582.129216] LNet: 83427:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1582.752897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1582.832114] LNet: Removed LNI 192.168.204.5@tcp231 [ 1582.852994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1582.892523] LNet: Added LNI 192.168.204.5@tcp231 [8/256/0/60] [ 1584.249258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1584.292754] LNet: Removed LNI 192.168.204.5@tcp231 [ 1584.316962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.446161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1584.478464] LNet: Added LNI 192.168.204.5@tcp231 [8/256/0/180] [ 1584.483486] LNet: 83578:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1585.073355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1585.116178] LNet: Removed LNI 192.168.204.5@tcp231 [ 1585.159652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1585.193438] LNet: Added LNI 192.168.204.5@tcp231 [8/256/0/180] [ 1594.706256] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 14:24:51 (1782757491) [ 1595.070083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1595.124921] LNet: Removed LNI 192.168.204.5@tcp231 [ 1595.182239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.272964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1595.309948] LNet: Added LNI 192.168.204.5@tcp232 [8/256/0/180] [ 1595.912871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1595.961762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1595.993288] LNet: Removed LNI 192.168.204.5@tcp232 [ 1596.018259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1596.072219] LNet: Added LNI 192.168.204.5@tcp232 [8/256/0/180] [ 1596.989695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1597.029124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1597.069110] LNet: Removed LNI 192.168.204.5@tcp232 [ 1597.098957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1597.139681] LNet: Added LNI 192.168.204.5@tcp232 [8/256/0/180] [ 1597.831405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1597.879110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1597.920491] LNet: Removed LNI 192.168.204.5@tcp232 [ 1597.948243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1597.987481] LNet: Added LNI 192.168.204.5@tcp232 [8/256/0/180] [ 1598.772540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1598.839494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1598.892477] LNet: Removed LNI 192.168.204.5@tcp232 [ 1598.909640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1605.941626] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 14:25:02 (1782757502) [ 1607.106474] Key type .llcrypt unregistered [ 1607.116214] Key type ._llcrypt unregistered [ 1614.233911] Key type ._llcrypt registered [ 1614.239513] Key type .llcrypt registered [ 1614.309365] LNet: 85656:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1614.309365] mask usage: [+|-] ... [ 1614.493203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1614.518714] alg: No test for adler32 (adler32-zlib) [ 1615.521132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1615.557230] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1623.563734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.105@tcp [ 1633.678893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.715189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1633.740540] LNet: Added LNI 192.168.204.5@tcp1 [8/256/0/180] [ 1635.018778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_interrupt -r 2 [ 1635.058167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp1 -m GET -e local_interrupt -r 2 [ 1635.091963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_interrupt -r 2 [ 1635.126196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -m GET -e local_interrupt -r 2 [ 1635.169974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp -m GET -e local_interrupt -r 2 [ 1635.207153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp1 -m GET -e local_interrupt -r 2 [ 1635.236847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -m GET -e local_interrupt -r 2 [ 1635.276899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -m GET -e local_interrupt -r 2 [ 1635.320765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.5@tcp 192.168.204.105@tcp [ 1635.341541] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1635.348530] LNet: 1 local NIs in recovery (showing 1): 192.168.204.5@tcp [ 1635.592250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1635.625673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1635.656481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1635.680912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1635.718896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1635.749897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1635.777521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1635.809331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1635.828743] LNet: 86089:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1635.845839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1635.877344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1635.907968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1635.939280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1635.989660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1636.044433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1636.519628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_dropped -r 2 [ 1636.562925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp1 -m GET -e local_dropped -r 2 [ 1636.618125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_dropped -r 2 [ 1636.676670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -m GET -e local_dropped -r 2 [ 1636.715671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp -m GET -e local_dropped -r 2 [ 1636.765769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp1 -m GET -e local_dropped -r 2 [ 1636.794434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -m GET -e local_dropped -r 2 [ 1636.830946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -m GET -e local_dropped -r 2 [ 1636.873690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.5@tcp 192.168.204.105@tcp [ 1636.898698] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1637.147282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1637.177122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1637.202491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1637.230506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1637.261296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1637.294378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1637.329860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1637.365616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1637.383436] LNet: 86209:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1637.409199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1637.451687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1637.488746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1637.534747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1637.595898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1637.627369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1638.137589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_aborted -r 2 [ 1638.179498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp1 -m GET -e local_aborted -r 2 [ 1638.234922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_aborted -r 2 [ 1638.287399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -m GET -e local_aborted -r 2 [ 1638.340029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp -m GET -e local_aborted -r 2 [ 1638.382511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp1 -m GET -e local_aborted -r 2 [ 1638.427250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -m GET -e local_aborted -r 2 [ 1638.466753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -m GET -e local_aborted -r 2 [ 1638.491187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.5@tcp 192.168.204.105@tcp [ 1638.511341] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1638.717619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1638.743808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1638.771765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1638.799776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1638.825699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1638.854973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1638.891984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1638.921413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1638.947721] LNet: 86330:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1638.970486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1638.999369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1639.028307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1639.057258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1639.097959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1639.143869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1639.652779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_no_route -r 2 [ 1639.690079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp1 -m GET -e local_no_route -r 2 [ 1639.721686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_no_route -r 2 [ 1639.775794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -m GET -e local_no_route -r 2 [ 1639.824484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp -m GET -e local_no_route -r 2 [ 1639.885847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp1 -m GET -e local_no_route -r 2 [ 1639.942456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -m GET -e local_no_route -r 2 [ 1640.008842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -m GET -e local_no_route -r 2 [ 1640.063741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.5@tcp 192.168.204.105@tcp [ 1640.313458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1640.345342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1640.374961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1640.400600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1640.426950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1640.466476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1640.512476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1640.551128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1640.575593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1640.602858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1640.654582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1640.688106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1640.727936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1640.767114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1641.179808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e local_timeout -r 2 [ 1641.211378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp1 -m GET -e local_timeout -r 2 [ 1641.250666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e local_timeout -r 2 [ 1641.318828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -m GET -e local_timeout -r 2 [ 1641.363371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp -m GET -e local_timeout -r 2 [ 1641.407802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp1 -m GET -e local_timeout -r 2 [ 1641.448404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -m GET -e local_timeout -r 2 [ 1641.515903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -m GET -e local_timeout -r 2 [ 1641.585727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.5@tcp 192.168.204.105@tcp [ 1641.608729] LNet: There was an unexpected network error while writing to 192.168.204.105: rc = -22 [ 1641.621219] LNet: Skipped 1 previous similar message [ 1641.941405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1641.983757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1642.012540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1642.065343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1642.117141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1642.151586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1642.195203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1642.265900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1642.313373] LNet: 86570:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1642.330216] LNet: 86570:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1642.353775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1642.426152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1642.476951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1642.545039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1642.639182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1642.701702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1643.372201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp -m GET -e remote_dropped -r 2 [ 1643.428820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.105@tcp1 -m GET -e remote_dropped -r 2 [ 1643.505903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp -m GET -e remote_dropped -r 2 [ 1643.607639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp -d 192.168.204.5@tcp1 -m GET -e remote_dropped -r 2 [ 1643.686806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp -m GET -e remote_dropped -r 2 [ 1643.731049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.105@tcp1 -m GET -e remote_dropped -r 2 [ 1643.811656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp -m GET -e remote_dropped -r 2 [ 1643.860157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.5@tcp1 -d 192.168.204.5@tcp1 -m GET -e remote_dropped -r 2 [ 1643.956790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.5@tcp 192.168.204.105@tcp [ 1643.999810] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.105@tcp [ 1644.258408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1644.303735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1644.360057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1644.391285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1644.431385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1644.483589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1644.505812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1644.549826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1644.613242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1644.661552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1644.696218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1644.721628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1644.760234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1644.791220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1650.040205] LNet: 86764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1650.053350] LNetError: 86764:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1650.063039] LNet: Removed LNI 192.168.204.5@tcp [ 1650.077587] LNet: Removed LNI 192.168.204.5@tcp1 [ 1650.568238] Key type .llcrypt unregistered [ 1650.573268] Key type ._llcrypt unregistered [ 1668.808161] Key type ._llcrypt registered [ 1668.813147] Key type .llcrypt registered [ 1670.018513] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 14:26:06 (1782757566) [ 1670.378816] LNet: 87571:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1670.378816] mask usage: [+|-] ... [ 1670.559232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1670.571609] alg: No test for adler32 (adler32-zlib) [ 1671.488347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1671.536235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1671.571122] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1672.523617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1672.580925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.5@tcp [ 1672.598721] LNetError: 87743:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.5@tcp to recovery [ 1672.721570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.5@tcp [ 1672.888287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1680.819758] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 14:26:17 (1782757577) [ 1680.948206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1680.976952] LNet: 88329:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1681.003917] LNetError: 88329:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1681.040885] LNet: Removed LNI 192.168.204.5@tcp [ 1681.098378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.204424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1681.242281] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1682.110877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1682.125627] LNet: Added LNI 192.168.204.5@tcp1 [8/256/0/180] [ 1682.976384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.5@tcp [ 1683.007070] LNetError: 88600:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.5@tcp to recovery [ 1683.232271] LNet: 1 local NIs in recovery (showing 1): 192.168.204.5@tcp [ 1683.244582] LNet: There was an unexpected network error while writing to 192.168.204.5: rc = -22 [ 1684.334531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.5@tcp [ 1695.091464] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 14:26:30 (1782757590) [ 1697.874984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg405-server [ 1700.741358] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 14:26:37 (1782757597) [ 1703.797036] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg405-server [ 1707.169967] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 14:26:43 (1782757603) [ 1709.956944] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg405-server [ 1712.109894] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 14:26:48 (1782757608) [ 1712.266872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.308826] LNet: 90219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.328315] LNetError: 90219:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.339207] LNet: Removed LNI 192.168.204.5@tcp [ 1712.379186] LNet: Removed LNI 192.168.204.5@tcp1 [ 1712.417733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.458512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1712.486581] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1713.278308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1713.422291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1713.507385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1713.661158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1713.740728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1713.826823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1721.718753] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 14:26:58 (1782757618) [ 1722.129373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.162866] LNet: 90949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.169692] LNetError: 90949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.178759] LNetError: 90949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1722.198232] LNet: Removed LNI 192.168.204.5@tcp [ 1722.218230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.316165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1722.351575] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1734.882944] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 14:27:10 (1782757630) [ 1735.020228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.033522] LNet: 91656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.037590] LNetError: 91656:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1735.048924] LNet: Removed LNI 192.168.204.5@tcp [ 1735.064534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.174786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1735.202246] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1744.206040] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 14:27:20 (1782757640) [ 1746.410418] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1748.155720] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 14:27:25 (1782757645) [ 1748.655266] LNet: 92698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1748.660947] LNetError: 92698:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1748.674029] LNet: Removed LNI 192.168.204.5@tcp [ 1749.591495] Key type .llcrypt unregistered [ 1749.595792] Key type ._llcrypt unregistered [ 1756.072155] Key type ._llcrypt registered [ 1756.077223] Key type .llcrypt registered [ 1756.146956] LNet: 92944:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1756.146956] mask usage: [+|-] ... [ 1756.295930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1756.308120] alg: No test for adler32 (adler32-zlib) [ 1757.353787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1757.374493] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1769.487203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.105@tcp [ 1775.931522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.105@tcp [ 1776.337385] LNet: 93133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1776.348344] LNetError: 93133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.370453] LNet: Removed LNI 192.168.204.5@tcp [ 1776.789416] Key type .llcrypt unregistered [ 1776.791608] Key type ._llcrypt unregistered [ 1791.151416] Key type ._llcrypt registered [ 1791.156688] Key type .llcrypt registered [ 1792.843941] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1795.735461] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1798.092722] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 14:28:14 (1782757694) [ 1800.377023] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1802.909283] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 14:28:19 (1782757699) [ 1804.970859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg405-server [ 1806.977843] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 14:28:23 (1782757703) [ 1809.502735] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg405-server [ 1812.285508] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 14:28:28 (1782757708) [ 1813.697567] LNet: 95182:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1813.697567] mask usage: [+|-] ... [ 1814.000605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1814.088694] alg: No test for adler32 (adler32-zlib) [ 1815.426312] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 1815.776336] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1817.520200] Key type lgssc registered [ 1819.056497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1840.005338] Key type lgssc unregistered [ 1840.285845] LNet: 96313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.299495] LNetError: 96313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.311646] LNet: Removed LNI 192.168.204.5@tcp [ 1841.170474] Key type .llcrypt unregistered [ 1841.176434] Key type ._llcrypt unregistered [ 1855.895865] Key type ._llcrypt registered [ 1855.899486] Key type .llcrypt registered [ 1857.508802] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 14:29:14 (1782757754) [ 1857.844186] LNet: 97108:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1857.844186] mask usage: [+|-] ... [ 1858.029714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1858.039605] alg: No test for adler32 (adler32-zlib) [ 1859.002394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1859.137304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1859.204615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1859.298672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1859.350056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1871.067626] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 14:29:26 (1782757766) [ 1871.201893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1871.232628] LNet: 97722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1871.279787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.384168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1871.436187] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1872.722874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.762214] LNet: 97873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.771671] LNetError: 97873:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.786964] LNet: Removed LNI 192.168.204.5@tcp [ 1872.799704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.853457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1872.875388] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1873.183206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1873.225560] LNet: Removed LNI 192.168.204.5@tcp [ 1873.247923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1873.301101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1873.328751] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1882.439939] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 14:29:39 (1782757779) [ 1882.553545] LNet: 98485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.564850] LNet: 98485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1882.588513] LNetError: 98485:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.595625] LNetError: 98485:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1882.619610] LNet: Removed LNI 192.168.204.5@tcp [ 1883.525386] Key type .llcrypt unregistered [ 1883.528542] Key type ._llcrypt unregistered [ 1889.548875] Key type ._llcrypt registered [ 1889.551986] Key type .llcrypt registered [ 1889.597630] LNet: 98725:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1889.597630] mask usage: [+|-] ... [ 1889.817516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1889.838851] alg: No test for adler32 (adler32-zlib) [ 1890.869578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1891.042586] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1891.056286] LNet: *** cfs_fail_loc=e003, val=100*** [ 1891.826936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 1891.852527] LNet: *** cfs_fail_loc=e003, val=100*** [ 1891.864635] LNet: Skipped 2 previous similar messages [ 1891.881434] Ping reply received with mapping: [ 1891.883366] found metadata reply NI[192.168.204.5@tcp] [ 1891.886319] 05 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1891.890542] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1891.893665] 5a 5a 5a 5a 5a 5a 5a 5a [ 1891.991133] LNet: 98892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1891.996491] LNetError: 98892:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.031295] LNet: Removed LNI 192.168.204.5@tcp [ 1893.734497] Key type .llcrypt unregistered [ 1893.739125] Key type ._llcrypt unregistered [ 1906.582476] Key type ._llcrypt registered [ 1906.585650] Key type .llcrypt registered [ 1908.267909] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 14:30:04 (1782757804) [ 1908.899494] Key type .llcrypt unregistered [ 1908.909872] Key type ._llcrypt unregistered [ 1916.621679] Key type ._llcrypt registered [ 1916.625298] Key type .llcrypt registered [ 1916.660765] LNet: 99889:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1916.660765] mask usage: [+|-] ... [ 1916.883477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1916.905154] alg: No test for adler32 (adler32-zlib) [ 1917.973239] LNetError: 99939:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1918.302521] Key type .llcrypt unregistered [ 1918.306291] Key type ._llcrypt unregistered [ 1924.230247] Key type ._llcrypt registered [ 1924.235460] Key type .llcrypt registered [ 1925.855656] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 14:30:22 (1782757822) [ 1926.717130] Key type .llcrypt unregistered [ 1926.719673] Key type ._llcrypt unregistered [ 1934.725459] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1934.731169] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1937.354454] Key type ._llcrypt registered [ 1937.356648] Key type .llcrypt registered [ 1937.406628] LNet: 100808:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1937.406628] mask usage: [+|-] ... [ 1937.682693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1937.699853] alg: No test for adler32 (adler32-zlib) [ 1938.667698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.740850] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1948.227627] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 14:30:45 (1782757845) [ 1948.511853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1948.542760] LNet: 101721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.552520] LNetError: 101721:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1948.569858] LNet: Removed LNI 10.1.2.3@tcp [ 1948.607291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.341380] Key type .llcrypt unregistered [ 1949.346586] Key type ._llcrypt unregistered [ 1955.240701] Key type ._llcrypt registered [ 1955.243703] Key type .llcrypt registered [ 1955.315413] LNet: 101952:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1955.315413] mask usage: [+|-] ... [ 1955.473314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1955.490677] alg: No test for adler32 (adler32-zlib) [ 1956.419096] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1956.502563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 1956.680918] LNet: 102009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.689544] LNetError: 102009:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1956.715427] LNet: Removed LNI 192.168.204.5@tcp [ 1957.332320] Key type .llcrypt unregistered [ 1957.339763] Key type ._llcrypt unregistered [ 1969.759172] Key type ._llcrypt registered [ 1969.760747] Key type .llcrypt registered [ 1971.207500] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 14:31:07 (1782757867) [ 1972.315507] Key type .llcrypt unregistered [ 1972.318175] Key type ._llcrypt unregistered [ 1982.265193] Key type ._llcrypt registered [ 1982.269751] Key type .llcrypt registered [ 1982.412312] LNet: 102999:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1982.412312] mask usage: [+|-] ... [ 1982.667540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1982.705579] alg: No test for adler32 (adler32-zlib) [ 1992.014542] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 14:31:27 (1782757887) [ 1992.224295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.284474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.415369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1992.447587] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 1993.702063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1993.781263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1993.839748] LNet: Removed LNI 192.168.204.5@tcp [ 1994.144688] LNet: 103761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1994.746709] Key type .llcrypt unregistered [ 1994.757941] Key type ._llcrypt unregistered [ 2009.559273] Key type ._llcrypt registered [ 2009.563510] Key type .llcrypt registered [ 2011.153971] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 14:31:47 (1782757907) [ 2011.576298] LNet: 104527:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2011.576298] mask usage: [+|-] ... [ 2011.880610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2011.900469] alg: No test for adler32 (adler32-zlib) [ 2012.903695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2012.977525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2013.011835] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2014.130366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 2014.200544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.5@tcp [ 2014.244507] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2014.250498] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2014.255321] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2014, 0, 0 [ 2014.261707] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2014.265527] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2014.270036] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2014.272828] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2014, 0, 0 [ 2014.276261] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2014.279161] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2014.282661] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2014.285696] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2014, 0, 2064 [ 2014.289795] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2014.292807] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2014.296838] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2014.302895] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2014, 0, 2064 [ 2014.307450] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2021.562930] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 14:31:58 (1782757918) [ 2022.173059] LNet: 105302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2022.183262] LNetError: 105302:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2022.203809] LNet: Removed LNI 192.168.204.5@tcp [ 2022.869318] Key type .llcrypt unregistered [ 2022.873516] Key type ._llcrypt unregistered [ 2030.250223] Key type ._llcrypt registered [ 2030.256197] Key type .llcrypt registered [ 2030.320922] LNet: 105526:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2030.320922] mask usage: [+|-] ... [ 2030.525348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2030.551681] alg: No test for adler32 (adler32-zlib) [ 2031.436807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2031.476533] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2040.239759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.105@tcp [ 2051.221663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2051.271631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2051.292666] LNet: Added LNI 192.168.204.5@tcp1 [8/256/0/180] [ 2052.826882] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2053.127252] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2053.137721] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2053.145706] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2055.654588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2055.701346] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2068.321803] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 14:32:45 (1782757965) [ 2068.671442] LNet: 106811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.680610] LNetError: 106811:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.692315] LNet: Removed LNI 192.168.204.5@tcp [ 2069.728224] LNet: Removed LNI 192.168.204.5@tcp1 [ 2069.730128] LNetError: 106811:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2069.739311] LNetError: 106811:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2069.753650] LNet: Removed LNI 10.1.2.3@tcp99 [ 2070.582822] Key type .llcrypt unregistered [ 2070.588299] Key type ._llcrypt unregistered [ 2078.270173] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2078.282771] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2080.943446] Key type ._llcrypt registered [ 2080.946248] Key type .llcrypt registered [ 2081.021524] LNet: 107078:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2081.021524] mask usage: [+|-] ... [ 2081.291753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2081.315854] alg: No test for adler32 (adler32-zlib) [ 2082.303700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2082.383458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2082.411429] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2083.120967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2083.142849] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2085.761863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2085.793576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2085.825912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2085.847256] LNetError: 107557:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2085.893768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2085.932475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2086.284715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2086.375541] LNet: 107577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.383384] LNetError: 107577:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2086.387896] LNet: Removed LNI 10.1.2.3@tcp [ 2087.400660] LNet: Removed LNI 192.168.204.5@tcp [ 2088.401637] Key type .llcrypt unregistered [ 2088.407127] Key type ._llcrypt unregistered [ 2105.051511] Key type ._llcrypt registered [ 2105.055582] Key type .llcrypt registered [ 2107.172620] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 14:33:23 (1782758003) [ 2107.516430] LNet: 108377:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2107.516430] mask usage: [+|-] ... [ 2107.828710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2107.855600] alg: No test for adler32 (adler32-zlib) [ 2108.836300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.920936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2109.021620] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2110.183537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.5@tcp [ 2110.282584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg405-client@tcp [ 2117.452307] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 14:33:34 (1782758014) [ 2117.521472] LNet: 109106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.524856] LNetError: 109106:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2117.536275] LNet: Removed LNI 192.168.204.5@tcp [ 2118.204713] Key type .llcrypt unregistered [ 2118.209852] Key type ._llcrypt unregistered [ 2125.393269] Key type ._llcrypt registered [ 2125.395967] Key type .llcrypt registered [ 2125.697628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2125.722319] alg: No test for adler32 (adler32-zlib) [ 2126.659613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2126.738390] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2126.788723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2135.116854] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 14:33:51 (1782758031) [ 2137.313749] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2139.475454] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 14:33:56 (1782758036) [ 2141.856501] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2144.817039] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 14:34:01 (1782758041) [ 2144.930316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.988889] LNet: 110528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.998498] LNetError: 110528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2145.030817] LNet: Removed LNI 192.168.204.5@tcp [ 2145.069923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.203962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2145.234850] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2146.305757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2146.362808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2158.030878] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 14:34:13 (1782758053) [ 2158.149103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2158.174623] LNet: 111239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.179939] LNetError: 111239:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2158.199553] LNet: Removed LNI 192.168.204.5@tcp [ 2158.221849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.296176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2158.337688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2159.639453] Key type .llcrypt unregistered [ 2159.644694] Key type ._llcrypt unregistered [ 2178.431419] Key type ._llcrypt registered [ 2178.436797] Key type .llcrypt registered [ 2181.343348] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 14:34:36 (1782758076) [ 2181.665746] LNet: 112053:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2181.665746] mask usage: [+|-] ... [ 2181.981596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2182.006083] alg: No test for adler32 (adler32-zlib) [ 2183.144701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2183.325939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2183.380770] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2184.660476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2184.731704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2194.065256] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 14:34:50 (1782758090) [ 2194.262709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2194.290164] LNet: 112803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2194.300957] LNetError: 112803:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2194.328482] LNet: Removed LNI 192.168.204.5@tcp [ 2194.381059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2194.455976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2194.511098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2202.119426] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 14:34:58 (1782758098) [ 2202.222144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2202.280357] LNet: 113381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2202.325965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2202.437143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2203.001842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2203.041333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2203.152946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2203.702346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2203.733744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2203.810709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2203.834651] LNet: 113419:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.854889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.922977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2204.617640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2204.657044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2204.740905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2205.138727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2205.162154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2214.188961] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 14:35:10 (1782758110) [ 2214.733781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2214.774842] LNet: 114020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.802760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2214.849640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2214.866254] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2215.823475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2215.851800] LNet: Added LNI 192.168.204.5@tcp2 [8/256/0/180] [ 2217.681487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2217.844259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2218.346420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2218.459217] LNetError: 114299:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2218.505790] LNet: Removed LNI 192.168.204.5@tcp [ 2218.540661] LNet: Removed LNI 192.168.204.5@tcp2 [ 2218.619738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2218.794585] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2218.804648] LNet: Added LNI 192.168.204.5@tcp2 [8/256/0/180] [ 2218.819699] LNet: 114305:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2219.966135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2220.207734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2220.224830] LNet: 114343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.231257] LNet: 114343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2220.245484] LNetError: 114343:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.263421] LNetError: 114343:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2220.273764] LNet: Removed LNI 192.168.204.5@tcp [ 2220.289910] LNet: Removed LNI 192.168.204.5@tcp2 [ 2220.315942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2220.375629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2220.410344] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2221.210620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2221.229550] LNet: Added LNI 192.168.204.5@tcp2 [8/256/0/180] [ 2222.678597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2222.740562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2222.794644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2222.982337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2222.999432] LNetError: 114627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2223.004791] LNetError: 114627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2224.033566] LNet: Removed LNI 192.168.204.5@tcp [ 2225.064050] LNet: Removed LNI 192.168.204.5@tcp2 [ 2225.120932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.167279] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2225.172822] LNet: Added LNI 192.168.204.5@tcp2 [8/256/0/180] [ 2225.176950] LNet: 114633:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2226.084776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2226.411295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.438617] LNet: Removed LNI 192.168.204.5@tcp [ 2227.499742] LNet: Removed LNI 192.168.204.5@tcp2 [ 2227.546307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.672065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2227.708330] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2228.765127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2228.788218] LNet: Added LNI 192.168.204.5@tcp2 [8/256/0/180] [ 2229.977901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2230.087691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2230.436505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2230.472836] LNet: 114950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2230.481863] LNet: 114950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2230.494963] LNetError: 114950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2230.503888] LNetError: 114950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2231.523566] LNet: Removed LNI 192.168.204.5@tcp [ 2231.553386] LNet: Removed LNI 192.168.204.5@tcp2 [ 2231.623491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2231.787736] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2231.801977] LNet: Added LNI 192.168.204.5@tcp2 [8/256/0/180] [ 2231.813747] LNet: 114956:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2232.377842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2232.748824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2233.824326] LNet: Removed LNI 192.168.204.5@tcp [ 2233.849213] LNet: Removed LNI 192.168.204.5@tcp2 [ 2233.900815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2233.958319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2233.992680] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2235.010722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2235.029349] LNet: Added LNI 192.168.204.5@tcp2 [8/256/0/180] [ 2236.338651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2236.428420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2236.505769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2237.003758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2237.051471] LNet: Removed LNI 192.168.204.5@tcp [ 2238.129424] LNet: Removed LNI 192.168.204.5@tcp2 [ 2238.221296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2238.352987] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2238.365188] LNet: Added LNI 192.168.204.5@tcp2 [8/256/0/180] [ 2238.377937] LNet: 115285:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2239.290133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2239.579406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2239.623754] LNetError: 115322:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2239.635667] LNetError: 115322:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2239.654464] LNet: Removed LNI 192.168.204.5@tcp [ 2240.684164] LNet: Removed LNI 192.168.204.5@tcp2 [ 2240.760715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2240.877702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2240.918933] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2241.787794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2241.815442] LNet: Added LNI 192.168.204.5@tcp2 [8/256/0/180] [ 2243.432418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2243.551043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2243.660865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2244.336886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2245.408165] LNet: Removed LNI 192.168.204.5@tcp [ 2245.432964] LNet: Removed LNI 192.168.204.5@tcp2 [ 2245.489911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2245.627870] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2245.666742] LNet: Added LNI 192.168.204.5@tcp2 [8/256/0/180] [ 2245.686774] LNet: 115625:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2246.723387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2247.255408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2247.292764] LNet: 115677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2247.301799] LNet: 115677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2248.354550] LNet: Removed LNI 192.168.204.5@tcp [ 2248.374112] LNet: Removed LNI 192.168.204.5@tcp2 [ 2248.429712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.591607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2248.641339] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2250.002201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2250.050759] LNet: Added LNI 192.168.204.5@tcp2 [8/256/0/180] [ 2251.705664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2251.762677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2251.819743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2251.903764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2252.501599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2253.536753] LNet: Removed LNI 192.168.204.5@tcp [ 2254.568210] LNet: Removed LNI 192.168.204.5@tcp2 [ 2254.611339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.730281] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2254.750903] LNet: Added LNI 192.168.204.5@tcp2 [8/256/0/180] [ 2254.765470] LNet: 115986:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2255.399471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2263.903645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2266.325219] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 14:36:02 (1782758162) [ 2266.493175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2266.528169] LNetError: 116741:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2266.538211] LNetError: 116741:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2266.553194] LNet: Removed LNI 192.168.204.5@tcp [ 2267.635186] LNet: Removed LNI 192.168.204.5@tcp2 [ 2267.675577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2267.995802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2268.026414] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2268.900740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2268.923971] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2270.236669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.269751] LNet: Removed LNI 10.1.2.3@tcp [ 2270.282552] LNet: Removed LNI 192.168.204.5@tcp [ 2270.300367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2270.370168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2270.428095] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2270.435154] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2271.263049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2271.309061] LNet: Removed LNI 192.168.204.5@tcp [ 2271.319486] LNet: Removed LNI 10.1.2.3@tcp [ 2281.787572] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 14:36:17 (1782758177) [ 2281.902361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2281.929883] LNet: 117615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.933922] LNet: 117615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2282.017772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.236287] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2282.535428] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2282.563886] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2282.580234] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2285.057985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2285.188071] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2286.600290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2286.661699] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2289.859607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.5@tcp [ 2289.997626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2290.206696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.5@tcp -m PUT -e local_timeout -r 1 [ 2290.410529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2290.606631] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2290.621925] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2291.356942] LNetError: 117686:0:(lib-move.c:4714:lnet_parse()) 192.168.204.5@tcp, src 192.168.204.5@tcp: Dropping PUT (error -108 looking up sender) [ 2292.384967] LNet: Removed LNI 10.1.2.3@tcp [ 2293.417990] LNet: Removed LNI 192.168.204.5@tcp [ 2294.413435] Key type .llcrypt unregistered [ 2294.416635] Key type ._llcrypt unregistered [ 2315.350042] Key type ._llcrypt registered [ 2315.353308] Key type .llcrypt registered [ 2316.947986] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 14:36:53 (1782758213) [ 2317.183187] LNet: 118919:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2317.183187] mask usage: [+|-] ... [ 2317.325651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2317.361257] alg: No test for adler32 (adler32-zlib) [ 2318.375157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2318.757669] LNet: 119000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2319.516409] Key type .llcrypt unregistered [ 2319.520654] Key type ._llcrypt unregistered [ 2326.417490] Key type ._llcrypt registered [ 2326.421414] Key type .llcrypt registered [ 2326.475467] LNet: 119229:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2326.475467] mask usage: [+|-] ... [ 2326.785215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2326.803438] alg: No test for adler32 (adler32-zlib) [ 2327.756720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2327.810133] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2336.324541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.105@tcp [ 2336.580539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2337.263481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.105@tcp [ 2337.582990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2337.636733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2337.678222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2337.728891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2337.777628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2337.828366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2337.872802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2337.904637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2337.932668] LNet: 119450:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2337.949199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2337.975612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2337.996598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2338.028512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2338.075396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2338.080333] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.105@tcp [ 2338.130491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2338.887912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2339.125625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2339.171947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2339.214627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2339.263139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2339.309543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2339.342484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2339.393166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2339.437972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2339.455082] LNet: 119567:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2339.471078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2339.499515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2339.528292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2339.567956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2339.609325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2339.637941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2340.372505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.105@tcp [ 2340.636380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2340.695715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2340.757401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2340.814754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2340.874889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2340.923501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2340.970932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2341.003179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2341.032925] LNet: 119684:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2341.065497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2341.110940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2341.161052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2341.200560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2341.239395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2341.274943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2342.027415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2342.377781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2342.419577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2342.474203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2342.521103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2342.572342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2342.627790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2342.671696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2342.717444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2342.757223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2342.789166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2342.822724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2342.853958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2342.895573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2342.948870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2343.415871] LNet: 119848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2343.422297] LNetError: 119848:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.486857] LNet: Removed LNI 192.168.204.5@tcp [ 2344.979786] Key type .llcrypt unregistered [ 2344.985118] Key type ._llcrypt unregistered [ 2365.487694] Key type ._llcrypt registered [ 2365.497218] Key type .llcrypt registered [ 2367.754808] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 14:37:43 (1782758263) [ 2368.291755] LNet: 120654:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2368.291755] mask usage: [+|-] ... [ 2368.497944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2368.546548] alg: No test for adler32 (adler32-zlib) [ 2369.586846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2370.230968] LNet: 120736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2371.419401] Key type .llcrypt unregistered [ 2371.426871] Key type ._llcrypt unregistered [ 2380.886512] Key type ._llcrypt registered [ 2380.889726] Key type .llcrypt registered [ 2380.927320] LNet: 120965:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2380.927320] mask usage: [+|-] ... [ 2381.164390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2381.199106] alg: No test for adler32 (adler32-zlib) [ 2382.228202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2382.276455] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2391.688498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.105@tcp [ 2403.609946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2403.664300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2403.679706] LNet: Added LNI 192.168.204.5@tcp1 [8/256/0/180] [ 2404.956525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2405.835818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.105@tcp [ 2406.121830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2406.159406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2406.196869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2406.240834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2406.290956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2406.339715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2406.383921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2406.426893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2406.464231] LNet: 121389:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2406.505375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2406.568951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2406.618960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2406.670910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2406.713100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2406.757303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2407.789160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2407.906691] LNet: 2 peer NIs in recovery (showing 2): 192.168.204.105@tcp, 192.168.204.105@tcp1 [ 2408.122294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2408.163001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2408.249862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2408.313967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2408.392419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2408.488255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2408.554161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2408.626819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2408.666683] LNet: 121506:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2408.695860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2408.838445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2408.907371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2408.965737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2409.022282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2409.064564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2409.967472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.105@tcp [ 2410.375820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2410.414985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2410.449422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2410.468882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2410.487980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2410.514788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2410.534422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2410.552227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2410.564094] LNet: 121624:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2410.579635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2410.604176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2410.635913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2410.685751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2410.723881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2410.786265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2411.337920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2411.738633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2411.786390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2411.813215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2411.871786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2411.924246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2411.978729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2412.059306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2412.121666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2412.192972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2412.248292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2412.327178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2412.377891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2412.441163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2412.516170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2419.145373] LNet: 121822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2419.165757] LNetError: 121822:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2419.180941] LNet: Removed LNI 192.168.204.5@tcp [ 2419.196920] LNet: Removed LNI 192.168.204.5@tcp1 [ 2419.648710] Key type .llcrypt unregistered [ 2419.650472] Key type ._llcrypt unregistered [ 2436.537473] Key type ._llcrypt registered [ 2436.544473] Key type .llcrypt registered [ 2438.059800] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 14:38:54 (1782758334) [ 2445.554468] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 14:39:02 (1782758342) [ 2452.732900] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 14:39:09 (1782758349) [ 2460.053237] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 14:39:17 (1782758357) [ 2467.732515] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 14:39:24 (1782758364) [ 2475.076664] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 14:39:31 (1782758371) [ 2484.299235] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 14:39:41 (1782758381) [ 2493.717679] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 14:39:50 (1782758390) [ 2502.716718] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 14:39:59 (1782758399) [ 2512.858927] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 14:40:09 (1782758409) [ 2521.310826] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 14:40:18 (1782758418) [ 2528.766613] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 14:40:25 (1782758425) [ 2536.191518] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 14:40:33 (1782758433) [ 2544.808255] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 14:40:41 (1782758441) [ 2555.164906] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 14:40:51 (1782758451) [ 2563.673273] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 14:41:00 (1782758460) [ 2572.486282] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 14:41:08 (1782758468) [ 2580.972180] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 14:41:17 (1782758477) [ 2581.218869] LNet: 132361:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2581.218869] mask usage: [+|-] ... [ 2581.429844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2581.475321] alg: No test for adler32 (adler32-zlib) [ 2582.463488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2582.631416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2582.663468] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2583.958101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2583.980521] LNet: 132534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2583.989748] LNetError: 132534:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2584.016214] LNet: Removed LNI 192.168.204.5@tcp [ 2584.065952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2584.130413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2584.177896] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2592.191689] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 14:41:28 (1782758488) [ 2600.456896] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 14:41:37 (1782758497) [ 2608.541807] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 14:41:45 (1782758505) [ 2616.898736] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 14:41:53 (1782758513) [ 2624.511902] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 14:42:01 (1782758521) [ 2631.607876] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 14:42:08 (1782758528) [ 2638.898723] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 14:42:15 (1782758535) [ 2645.638297] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 14:42:22 (1782758542) [ 2653.319312] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 14:42:30 (1782758550) [ 2663.750428] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 14:42:40 (1782758560) [ 2671.378791] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 14:42:48 (1782758568) [ 2679.112985] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 14:42:56 (1782758576) [ 2685.909893] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2486 sec ========== 14:43:02 (1782758582) [ 2686.584670] LNet: 140000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2686.590758] LNetError: 140000:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2686.604206] LNet: Removed LNI 192.168.204.5@tcp [ 2687.381822] Key type .llcrypt unregistered [ 2687.383722] Key type ._llcrypt unregistered [ 2701.497039] Key type ._llcrypt registered [ 2701.501033] Key type .llcrypt registered [ 2701.738591] LNet: 140472:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2701.738591] mask usage: [+|-] ... [ 2701.977831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2702.004926] alg: No test for adler32 (adler32-zlib) [ 2703.177677] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 2703.482695] LNet: Added LNI 192.168.204.5@tcp [8/256/0/180] [ 2705.257301] Key type lgssc registered [ 2706.506830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2781.876950] Lustre: Mounted lustre-client [ 2782.111475] LNet: 141949:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2782.111475] mask usage: [+|-] ... [ 2788.183156] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2797.497765] Lustre: Unmounted lustre-client [ 2819.673915] Key type lgssc unregistered [ 2820.069072] LNet: 142932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2820.075048] LNetError: 142932:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2820.117537] LNet: Removed LNI 192.168.204.5@tcp [ 2820.816180] Key type .llcrypt unregistered [ 2820.820232] Key type ._llcrypt unregistered