[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 376956446 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003298] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.007862] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009008] pid_max: default: 32768 minimum: 301 [ 0.010136] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012026] SELinux: Initializing. [ 0.013060] *** VALIDATE selinux *** [ 0.021634] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026151] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027137] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028081] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029078] *** VALIDATE tmpfs *** [ 0.030427] *** VALIDATE proc *** [ 0.031186] *** VALIDATE cgroup *** [ 0.032005] *** VALIDATE cgroup2 *** [ 0.033215] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036022] Spectre V2 : User space: Vulnerable [ 0.037005] Speculative Store Bypass: Vulnerable [ 0.040617] debug: unmapping init [mem 0xffffffff98659000-0xffffffff98660fff] [ 0.042255] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043666] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044016] ... version: 2 [ 0.045009] ... bit width: 48 [ 0.046008] ... generic registers: 4 [ 0.047008] ... value mask: 0000ffffffffffff [ 0.048009] ... max period: 00007fffffffffff [ 0.049009] ... fixed-purpose events: 3 [ 0.050009] ... event mask: 000000070000000f [ 0.051258] rcu: Hierarchical SRCU implementation. [ 0.053486] smp: Bringing up secondary CPUs ... [ 0.054551] x86: Booting SMP configuration: [ 0.055013] .... node #0, CPUs: #1 #2 #3 [ 0.060227] smp: Brought up 1 node, 4 CPUs [ 0.062010] smpboot: Max logical packages: 1 [ 0.063006] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145020] node 0 deferred pages initialised in 80ms [ 0.150201] devtmpfs: initialized [ 0.151189] x86/mm: Memory block size: 128MB [ 0.153647] gcov: version magic: 0x41383552 [ 0.155208] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156039] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157244] pinctrl core: initialized pinctrl subsystem [ 0.159123] [ 0.159576] ************************************************************* [ 0.161006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162005] ** ** [ 0.164006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165005] ** ** [ 0.167006] ** This means that this kernel is built to expose internal ** [ 0.168004] ** IOMMU data structures, which may compromise security on ** [ 0.169006] ** your system. ** [ 0.171005] ** ** [ 0.172004] ** If you see this message and you are not debugging the ** [ 0.174005] ** kernel, report this immediately to your vendor! ** [ 0.175005] ** ** [ 0.176006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178006] ************************************************************* [ 0.179601] NET: Registered protocol family 16 [ 0.180302] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182026] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184029] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186084] cpuidle: using governor menu [ 0.187566] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.189390] PCI: Using configuration type 1 for base access [ 0.190078] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.196183] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.198038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.200086] cryptd: max_cpu_qlen set to 1000 [ 0.202350] ACPI: Added _OSI(Module Device) [ 0.203005] ACPI: Added _OSI(Processor Device) [ 0.203973] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.205006] ACPI: Added _OSI(Processor Aggregator Device) [ 0.207919] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.211377] ACPI: Interpreter enabled [ 0.212031] ACPI: PM: (supports S0 S3 S4 S5) [ 0.212987] ACPI: Using IOAPIC for interrupt routing [ 0.214069] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.216225] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.222871] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.225016] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.226007] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228038] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.230938] acpiphp: Slot [2] registered [ 0.232066] acpiphp: Slot [5] registered [ 0.233031] acpiphp: Slot [6] registered [ 0.233945] acpiphp: Slot [3] registered [ 0.235040] acpiphp: Slot [4] registered [ 0.235903] acpiphp: Slot [7] registered [ 0.237040] acpiphp: Slot [8] registered [ 0.237946] acpiphp: Slot [9] registered [ 0.239041] acpiphp: Slot [10] registered [ 0.239876] acpiphp: Slot [11] registered [ 0.240039] acpiphp: Slot [12] registered [ 0.240937] acpiphp: Slot [13] registered [ 0.242052] acpiphp: Slot [14] registered [ 0.242969] acpiphp: Slot [15] registered [ 0.244041] acpiphp: Slot [16] registered [ 0.245037] acpiphp: Slot [17] registered [ 0.246059] acpiphp: Slot [18] registered [ 0.246978] acpiphp: Slot [19] registered [ 0.247053] acpiphp: Slot [20] registered [ 0.248043] acpiphp: Slot [21] registered [ 0.249048] acpiphp: Slot [22] registered [ 0.251042] acpiphp: Slot [23] registered [ 0.252005] acpiphp: Slot [24] registered [ 0.252873] acpiphp: Slot [25] registered [ 0.253040] acpiphp: Slot [26] registered [ 0.253899] acpiphp: Slot [27] registered [ 0.255050] acpiphp: Slot [28] registered [ 0.255933] acpiphp: Slot [29] registered [ 0.257040] acpiphp: Slot [30] registered [ 0.257946] acpiphp: Slot [31] registered [ 0.258046] PCI host bridge to bus 0000:00 [ 0.258917] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.261008] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.262009] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.264009] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.266008] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.267009] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.269118] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.270632] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.273093] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.279579] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.282665] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.286012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.287009] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.289011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.292504] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.294618] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.296027] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.297616] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.301009] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.308012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.313011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.317862] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.323009] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.325010] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.337011] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.350369] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.355010] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.365012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.376010] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.382167] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.384331] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.387352] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.390302] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.392214] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.396169] iommu: Default domain type: Passthrough [ 0.398519] SCSI subsystem initialized [ 0.400127] ACPI: bus type USB registered [ 0.402073] usbcore: registered new interface driver usbfs [ 0.403033] usbcore: registered new interface driver hub [ 0.405057] usbcore: registered new device driver usb [ 0.406203] pps_core: LinuxPPS API ver. 1 registered [ 0.408010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.412047] PTP clock support registered [ 0.414070] EDAC MC: Ver: 3.0.0 [ 0.416119] PCI: Using ACPI for IRQ routing [ 0.417886] NetLabel: Initializing [ 0.419008] NetLabel: domain hash size = 128 [ 0.421008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.423063] NetLabel: unlabeled traffic allowed by default [ 0.425071] vgaarb: loaded [ 0.426091] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.428009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.434395] clocksource: Switched to clocksource kvm-clock [ 0.536217] VFS: Disk quotas dquot_6.6.0 [ 0.538730] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.540836] *** VALIDATE ramfs *** [ 0.541705] *** VALIDATE hugetlbfs *** [ 0.542702] pnp: PnP ACPI init [ 0.544653] pnp: PnP ACPI: found 6 devices [ 0.560543] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.564157] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.566665] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.568600] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.571408] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.573675] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.577028] NET: Registered protocol family 2 [ 0.579647] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.584662] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.588379] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.593984] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.597623] TCP: Hash tables configured (established 65536 bind 65536) [ 0.600285] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.602994] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.605262] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.607844] NET: Registered protocol family 1 [ 0.610177] RPC: Registered named UNIX socket transport module. [ 0.612121] RPC: Registered udp transport module. [ 0.613472] RPC: Registered tcp transport module. [ 0.614907] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.616877] NET: Registered protocol family 44 [ 0.618200] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.619961] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.621691] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.623663] PCI: CLS 0 bytes, default 64 [ 0.625097] Unpacking initramfs... [ 2.031246] debug: unmapping init [mem 0xffff9430fcc64000-0xffff9430fffcffff] [ 2.035727] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.038140] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.041138] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.524118] Initialise system trusted keyrings [ 2.525578] Key type blacklist registered [ 2.527215] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.535653] zbud: loaded [ 2.538220] *** VALIDATE nfs *** [ 2.539189] *** VALIDATE nfs4 *** [ 2.540478] pstore: using deflate compression [ 2.543558] Platform Keyring initialized [ 2.642549] NET: Registered protocol family 38 [ 2.644515] Key type asymmetric registered [ 2.649949] Asymmetric key parser 'x509' registered [ 2.651818] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.655238] io scheduler mq-deadline registered [ 2.656705] io scheduler kyber registered [ 2.658420] io scheduler bfq registered [ 2.661976] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.664652] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.667168] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.669756] ACPI: Power Button [PWRF] [ 2.674958] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.681107] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.692241] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.719739] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.751459] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.758346] Non-volatile memory driver v1.3 [ 2.759834] Linux agpgart interface v0.103 [ 2.791768] virtio_blk virtio1: [vda] 145160 512-byte logical blocks (74.3 MB/70.9 MiB) [ 2.795418] vda: detected capacity change from 0 to 74321920 [ 2.809983] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.812975] vdb: detected capacity change from 0 to 1073741824 [ 2.827515] libphy: Fixed MDIO Bus: probed [ 2.832630] usbcore: registered new interface driver usbserial_generic [ 2.835893] usbserial: USB Serial support registered for generic [ 2.837960] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.842114] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.843542] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.845931] mousedev: PS/2 mouse device common for all mice [ 2.849740] rtc_cmos 00:05: RTC can wake from S4 [ 2.852654] rtc_cmos 00:05: registered as rtc0 [ 2.854423] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.855678] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.861079] intel_pstate: CPU model not supported [ 2.864245] hid: raw HID events driver (C) Jiri Kosina [ 2.873813] usbcore: registered new interface driver usbhid [ 2.874454] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.877965] usbhid: USB HID core driver [ 2.878171] drop_monitor: Initializing network drop monitor service [ 2.884565] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.885694] Initializing XFRM netlink socket [ 2.886087] NET: Registered protocol family 10 [ 2.887961] Segment Routing with IPv6 [ 2.896100] NET: Registered protocol family 17 [ 2.897969] mpls_gso: MPLS GSO support [ 2.904481] RAS: Correctable Errors collector initialized. [ 2.906609] AVX version of gcm_enc/dec engaged. [ 2.908464] AES CTR mode by8 optimization enabled [ 2.987550] sched_clock: Marking stable (2987470090, 0)->(3789072078, -801601988) [ 2.991410] registered taskstats version 1 [ 2.994081] Loading compiled-in X.509 certificates [ 2.996403] zswap: loaded using pool lzo/zbud [ 3.020267] Key type big_key registered [ 3.032815] Key type encrypted registered [ 3.034928] ima: No TPM chip found, activating TPM-bypass! [ 3.037462] ima: Allocated hash algorithm: sha1 [ 3.039478] ima: No architecture policies found [ 3.041493] evm: Initialising EVM extended attributes: [ 3.043759] evm: security.selinux [ 3.045348] evm: security.ima [ 3.046760] evm: security.capability [ 3.048371] evm: HMAC attrs: 0x1 [ 3.051166] rtc_cmos 00:05: setting system clock to 2026-07-01 00:29:20 UTC (1782865760) [ 3.059186] debug: unmapping init [mem 0xffffffff99603000-0xffffffff997fffff] [ 3.062972] debug: unmapping init [mem 0xffffffff98382000-0xffffffff98658fff] [ 3.072194] Write protecting the kernel read-only data: 28672k [ 3.077450] debug: unmapping init [mem 0xffffffff96a03000-0xffffffff96bfffff] [ 3.080843] debug: unmapping init [mem 0xffffffff97314000-0xffffffff973fffff] [ 3.112919] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.122394] systemd[1]: Detected virtualization kvm. [ 3.124570] systemd[1]: Detected architecture x86-64. [ 3.126627] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.154865] systemd[1]: No hostname configured. [ 3.157546] systemd[1]: Set hostname to . [ 3.160410] random: systemd: uninitialized urandom read (16 bytes read) [ 3.162546] systemd[1]: Initializing machine ID from random generator. [ 3.229940] random: ln: uninitialized urandom read (6 bytes read) [ 3.326930] random: systemd: uninitialized urandom read (16 bytes read) [ 3.330243] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.341756] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.360422] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.105307] device-mapper: uevent: version 1.0.3 [ 4.107489] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 4.854378] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 4.886772] random: fast init done [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.109171] scsi host0: ata_piix [ 5.212047] scsi host1: ata_piix [ 5.213830] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.238175] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.557485] random: crng init done [ 9.559606] random: 7 urandom warning(s) missed due to ratelimiting [ 9.574607] dracut-initqueue[577]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.759901] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.953550] printk: systemd: 23 output lines suppressed due to ratelimiting [ 12.211033] SELinux: Disabled at runtime. [ 12.277944] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.286241] systemd[1]: Detected virtualization kvm. [ 12.287870] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.738964] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.743643] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.749041] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.755244] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.757871] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.765511] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.771241] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK [ 12.811482] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.235731] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.507205] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.508259] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.623226] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.634659] EDAC sbridge: Ver: 1.1.2 [ 14.703416] Key type dns_resolver registered [ 15.043880] NFS: Registering the id_resolver key type [ 15.045247] Key type id_resolver registered [ 15.046497] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg429-client login: [ 38.111202] libcfs: loading out-of-tree module taints kernel. [ 38.154069] Key type ._llcrypt registered [ 38.156311] Key type .llcrypt registered [ 38.421537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 38.426251] alg: No test for adler32 (adler32-zlib) [ 39.391799] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 39.678353] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 41.280180] Key type lgssc registered [ 41.826751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.367333] Lustre: Mounted lustre-client [ 87.633714] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 94.054683] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing check_logdir /tmp/testlogs/ [ 95.579780] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing yml_node [ 96.934967] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 97.746296] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 98.566295] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 99.097491] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Jun 30 20:30:56 EDT 2026 [ 103.709465] Lustre: Unmounted lustre-client [ 110.781940] Key type lgssc unregistered [ 110.900553] LNet: 8368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 110.903923] LNetError: 8368:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 110.911604] LNet: Removed LNI 192.168.204.29@tcp [ 111.183121] Key type .llcrypt unregistered [ 111.184891] Key type ._llcrypt unregistered [ 119.333918] Key type ._llcrypt registered [ 119.335704] Key type .llcrypt registered [ 119.349510] LNet: 8944:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 119.349510] mask usage: [+|-] ... [ 119.400250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 119.405455] alg: No test for adler32 (adler32-zlib) [ 120.202211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 120.224718] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 120.233771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 121.480675] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 121.514185] LNet: 9243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 121.516643] LNetError: 9243:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 121.523357] LNet: Removed LNI 192.168.204.29@tcp [ 121.734523] Key type .llcrypt unregistered [ 121.735694] Key type ._llcrypt unregistered [ 123.686893] Key type ._llcrypt registered [ 123.688010] Key type .llcrypt registered [ 124.106755] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 20:31:21 (1782865881) [ 124.300175] LNet: 9620:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 124.300175] mask usage: [+|-] ... [ 124.347039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 124.353303] alg: No test for adler32 (adler32-zlib) [ 125.151853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 125.186224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 125.195377] LNet: 9658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 125.202239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 125.220903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 125.228494] LNet: 9665:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 127.327408] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 20:31:24 (1782865884) [ 127.355874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 127.366425] LNet: 10237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 127.377112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 127.402638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 129.409955] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 20:31:26 (1782865886) [ 129.438189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 129.447027] LNet: 10821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 129.456186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 129.488296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 131.535948] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 20:31:28 (1782865888) [ 131.565808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 131.575312] LNet: 11403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 131.583628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 131.612516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 133.662121] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 20:31:30 (1782865890) [ 133.706994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 133.723103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 133.750679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 133.883714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 135.894393] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 20:31:32 (1782865892) [ 135.927033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 135.935901] LNet: 12578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 135.938185] LNet: 12578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 135.950671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 135.972904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 136.086327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 136.226672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 138.230135] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 20:31:35 (1782865895) [ 138.258852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 138.274585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 138.303195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 140.448135] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 20:31:37 (1782865897) [ 140.472821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 140.487752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 140.517795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 140.529179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 140.645248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 140.658498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 140.770785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 140.784199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 140.907648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 140.920489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 141.055050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 141.068394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 141.213778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 141.227100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 141.346374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 141.357092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 141.477072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 141.489499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 141.629868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 141.644176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 141.769505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 141.782636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 141.911413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 141.922788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 142.034894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 142.047224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 142.174469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 142.187854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 144.200226] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 20:31:41 (1782865901) [ 144.229941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 144.238170] LNet: 14436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 144.240108] LNet: 14436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 144.248362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 144.276831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 144.287941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 146.276385] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 20:31:43 (1782865903) [ 146.306410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 146.322683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 146.356475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 146.372325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 148.371026] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 20:31:45 (1782865905) [ 148.422203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 148.439481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 148.471865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 148.486475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 150.614368] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 20:31:47 (1782865907) [ 150.650271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 150.665249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 150.696408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 150.710728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 152.778577] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 20:31:49 (1782865909) [ 152.806775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 152.822518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 152.851671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 152.866198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 154.889858] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 20:31:51 (1782865911) [ 154.917936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 154.935177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 154.964895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 154.977546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 157.068515] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 20:31:54 (1782865914) [ 157.100316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 157.117638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 157.149808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 157.273251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 159.329351] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 20:31:56 (1782865916) [ 159.356170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 159.373424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 159.441797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 159.515111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 161.627884] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 20:31:58 (1782865918) [ 161.665585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 161.673516] LNet: 19165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 161.676307] LNet: 19165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 161.685191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 161.720099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 161.790793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 163.794178] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 20:32:00 (1782865920) [ 163.834042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 163.854458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 164.020039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 164.104744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 166.160939] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 20:32:03 (1782865923) [ 166.192339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 166.211280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 166.242436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 166.259439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 168.306137] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 20:32:05 (1782865925) [ 168.336372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 168.356608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 168.386772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 168.402540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 170.498955] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 20:32:07 (1782865927) [ 170.527619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 170.541644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 170.573524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 172.585292] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 20:32:09 (1782865929) [ 172.613752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 172.632735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 172.664068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 174.663876] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 20:32:11 (1782865931) [ 174.690855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 174.705795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 174.736293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 174.854535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 176.849804] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 20:32:13 (1782865933) [ 176.903064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 176.916781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 176.942157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 176.954415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 178.966937] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 20:32:15 (1782865935) [ 179.006825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 179.028128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 179.060932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 179.077771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 181.136463] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 20:32:18 (1782865938) [ 181.170106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 181.186974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 181.217138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 181.234754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 183.286440] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 20:32:20 (1782865940) [ 183.323653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 183.343440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 183.368522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 183.380919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 185.577485] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 20:32:22 (1782865942) [ 185.622283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 185.638668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 185.657300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 185.681093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 185.687817] LNetError: 25742:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 185.700555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 185.711413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 187.596605] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 20:32:24 (1782865944) [ 187.646916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 187.663845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 187.692769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KhPbqzpiaP [ 187.722924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KhPbqzpiaP [ 187.746822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KhPbqzpiaP [ 189.670093] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 20:32:26 (1782865946) [ 189.697208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 189.713877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 189.740613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 189.757568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 189.774405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 191.674621] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 20:32:28 (1782865948) [ 191.710280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 191.727198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 191.752720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 192.647796] LNet: Message forwarding will be enabled in 2s [ 195.744079] LNet: Message forwarding enabled [ 195.793621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 195.800969] LNet: Message forwarding will stop in 2s to 6s [ 199.951265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 200.459638] LNet: Message forwarding disabled [ 200.928489] LNet: Message forwarding will be enabled in 2s [ 202.908479] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 20:32:39 (1782865959) [ 202.948553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 202.956885] LNet: 28181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 202.959175] LNet: 28181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 203.468592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 203.493901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 205.412354] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 20:32:42 (1782865962) [ 205.453603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 205.471234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 205.491795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 205.503269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 205.514493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 205.526717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 205.538160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 205.549992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 205.561715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 205.574369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 205.586440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 205.597549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 205.608107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 205.619612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 205.632118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 205.642243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 205.652703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 205.663953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 205.674343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 205.686160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 205.697052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 205.703459] LNetError: 28807:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 207.591547] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 20:32:44 (1782865964) [ 207.634661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 207.649370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 207.672351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 209.706935] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 20:32:46 (1782865966) [ 210.274362] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg429-server [ 210.832370] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 20:32:47 (1782865967) [ 210.858459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 210.872647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 210.893865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 210.901955] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 211.393449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[30-34]@tcp [ 211.402100] LNet: 30520:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 213.409596] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 20:32:50 (1782865970) [ 213.965878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg429-server [ 214.496757] LNetError: 30315:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.31@tcp has gone from up to down [ 214.507288] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 20:32:51 (1782865971) [ 214.531312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 215.520203] LNetError: 30309:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.204.30@tcp [ 217.632099] LNetError: 30310:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.34@tcp has gone from up to down [ 217.635040] LNetError: 30310:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 217.637498] LNetError: 31448:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 217.648584] LNet: Removed LNI 192.168.204.29@tcp [ 217.656473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 217.675789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 217.684847] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 217.926250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[30-35/2]@tcp [ 217.936888] LNet: 31572:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 217.940137] LNet: 31572:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 217.948949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[30-35/2]@tcp [ 219.972159] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 20:32:56 (1782865976) [ 220.012342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.049172] LNetError: 32159:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 222.059472] LNet: Removed LNI 192.168.204.29@tcp [ 222.065931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.092467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 222.110614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.122511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.146478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 222.167665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.179829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.199513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 222.232334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.249417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.270000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 222.295516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.307280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.324647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 222.354603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.367767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.393181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 222.423264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.436396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.457139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 222.485544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.498503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.521669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 222.555480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.571464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.593595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 222.621330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.633969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.656682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 222.685682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.697892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.717083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 222.738361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.749394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.772166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 224.656937] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 20:33:01 (1782865981) [ 225.214805] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg429-server [ 225.739857] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 20:33:02 (1782865982) [ 226.332661] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg429-server [ 226.917363] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 20:33:03 (1782865983) [ 227.212239] Key type .llcrypt unregistered [ 227.214246] Key type ._llcrypt unregistered [ 229.334310] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 229.389195] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 229.394402] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 229.397421] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 231.601727] Key type ._llcrypt registered [ 231.602710] Key type .llcrypt registered [ 231.616878] LNet: 33839:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 231.616878] mask usage: [+|-] ... [ 231.677927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 231.682769] alg: No test for adler32 (adler32-zlib) [ 232.480801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 232.497669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 232.506397] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 232.750203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 232.757699] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 233.412819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 233.420886] LNet: Removed LNI 10.1.2.3@tcp [ 233.463716] LNet: 34280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 233.466749] LNetError: 34280:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 233.473265] LNet: Removed LNI 192.168.204.29@tcp [ 233.700452] Key type .llcrypt unregistered [ 233.701875] Key type ._llcrypt unregistered [ 237.310771] Key type ._llcrypt registered [ 237.312199] Key type .llcrypt registered [ 237.746637] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 20:33:14 (1782865994) [ 237.911355] Key type .llcrypt unregistered [ 237.912504] Key type ._llcrypt unregistered [ 239.839542] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 239.904901] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 239.906603] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 239.908663] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 242.100310] Key type ._llcrypt registered [ 242.101462] Key type .llcrypt registered [ 242.113927] LNet: 35314:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 242.113927] mask usage: [+|-] ... [ 242.168749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 242.174475] alg: No test for adler32 (adler32-zlib) [ 242.977655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 242.997572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 243.007023] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 243.250552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 243.259770] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 244.096387] LNet: 35771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 244.098691] LNetError: 35771:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 244.102982] LNet: Removed LNI 10.1.2.3@tcp [ 244.108346] LNet: Removed LNI 192.168.204.29@tcp [ 244.310495] Key type .llcrypt unregistered [ 244.312445] Key type ._llcrypt unregistered [ 247.881584] Key type ._llcrypt registered [ 247.883378] Key type .llcrypt registered [ 248.332228] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 20:33:25 (1782866005) [ 248.504433] Key type .llcrypt unregistered [ 248.505495] Key type ._llcrypt unregistered [ 250.423753] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 250.485350] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 250.489505] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 250.492708] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 252.703141] Key type ._llcrypt registered [ 252.704203] Key type .llcrypt registered [ 252.719295] LNet: 36806:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 252.719295] mask usage: [+|-] ... [ 252.776741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 252.782948] alg: No test for adler32 (adler32-zlib) [ 253.585583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 253.602893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 253.612500] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 254.220889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 254.228478] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 254.911547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 254.923828] LNet: Removed LNI 10.1.2.3@tcp [ 254.930769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 254.944525] LNet: Removed LNI 10.1.2.31@tcp [ 254.952028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 254.961029] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 255.216698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 255.225192] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 255.541260] LNet: 37688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 255.544237] LNetError: 37688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 255.546295] LNet: Removed LNI 10.1.2.31@tcp [ 255.554321] LNet: Removed LNI 10.1.2.3@tcp [ 255.773243] Key type .llcrypt unregistered [ 255.775168] Key type ._llcrypt unregistered [ 259.461967] Key type ._llcrypt registered [ 259.463966] Key type .llcrypt registered [ 259.917183] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 20:33:36 (1782866016) [ 259.987538] LNet: 38478:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 259.987538] mask usage: [+|-] ... [ 260.037120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 260.043087] alg: No test for adler32 (adler32-zlib) [ 260.842513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.859769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 260.867838] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 261.133265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 261.141382] LNet: 38639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 261.143591] LNetError: 38639:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 261.151277] LNet: Removed LNI 192.168.204.29@tcp [ 261.158755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 261.179959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.29@tcp [ 261.187928] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 261.428845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.29@tcp [ 261.441375] LNet: Removed LNI 192.168.204.29@tcp [ 261.669358] Key type .llcrypt unregistered [ 261.671353] Key type ._llcrypt unregistered [ 265.190972] Key type ._llcrypt registered [ 265.192018] Key type .llcrypt registered [ 265.632816] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 20:33:42 (1782866022) [ 266.191839] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg429-server [ 266.705343] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 20:33:43 (1782866023) [ 266.863339] Key type .llcrypt unregistered [ 266.864331] Key type ._llcrypt unregistered [ 268.792361] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 268.844682] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 268.849245] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 268.851373] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 271.032885] Key type ._llcrypt registered [ 271.033944] Key type .llcrypt registered [ 271.045189] LNet: 40144:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 271.045189] mask usage: [+|-] ... [ 271.094495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 271.100990] alg: No test for adler32 (adler32-zlib) [ 271.900583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.927508] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 272.171736] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 272.823747] LNet: 40603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 272.826848] LNetError: 40603:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 272.828919] LNet: Removed LNI 192.168.204.29@tcp1 [ 272.834294] LNet: Removed LNI 10.1.2.3@tcp2 [ 273.044287] Key type .llcrypt unregistered [ 273.045375] Key type ._llcrypt unregistered [ 276.437616] Key type ._llcrypt registered [ 276.438690] Key type .llcrypt registered [ 276.874574] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 20:33:53 (1782866033) [ 276.950170] LNet: 41405:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 276.950170] mask usage: [+|-] ... [ 277.007230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 277.013921] alg: No test for adler32 (adler32-zlib) [ 277.818785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 277.839029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 277.850738] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 278.154657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 278.165277] LNet: 41570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 278.167717] LNetError: 41570:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 278.178429] LNet: Removed LNI 192.168.204.29@tcp [ 278.185367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 278.206405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 278.215595] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 280.303863] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 20:33:57 (1782866037) [ 280.334411] LNet: 42153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 280.338892] LNetError: 42153:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 280.349814] LNet: Removed LNI 192.168.204.29@tcp [ 280.575543] Key type .llcrypt unregistered [ 280.576567] Key type ._llcrypt unregistered [ 282.431556] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 282.495094] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 282.496706] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 282.498755] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 284.718828] Key type ._llcrypt registered [ 284.719784] Key type .llcrypt registered [ 284.732173] LNet: 42436:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 284.732173] mask usage: [+|-] ... [ 284.782937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 284.788696] alg: No test for adler32 (adler32-zlib) [ 285.591391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 285.613242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 285.623393] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 285.874193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 285.882600] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 286.657372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 286.664903] LNet: 42902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 286.667285] LNetError: 42902:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 286.669308] LNet: Removed LNI 10.1.2.3@tcp [ 286.680364] LNet: Removed LNI 192.168.204.29@tcp [ 286.687717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 286.709772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 286.721153] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 286.722774] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 286.869612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 286.879052] LNet: Removed LNI 10.1.2.3@tcp [ 286.885350] LNet: Removed LNI 192.168.204.29@tcp [ 286.891524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 286.913765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 286.923145] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 286.925107] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 287.078854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 287.087351] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 287.438631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 287.445817] LNet: 43092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 287.447808] LNet: 43092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 287.449995] LNetError: 43092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 287.452113] LNetError: 43092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 287.454624] LNet: Removed LNI 10.1.2.3@tcp [ 287.455550] LNet: Removed LNI 192.168.204.29@tcp [ 287.465417] LNet: Removed LNI 192.168.204.29@tcp2 [ 287.472666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 287.494460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 287.503921] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 287.505419] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 287.507440] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 289.721585] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 20:34:06 (1782866046) [ 289.755787] LNet: 43680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 289.759238] LNetError: 43680:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 289.762897] LNetError: 43680:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 289.766488] LNet: Removed LNI 10.1.2.3@tcp [ 289.767611] LNet: Removed LNI 192.168.204.29@tcp [ 289.775887] LNet: Removed LNI 192.168.204.29@tcp2 [ 290.007370] Key type .llcrypt unregistered [ 290.008400] Key type ._llcrypt unregistered [ 291.886959] Key type ._llcrypt registered [ 291.887946] Key type .llcrypt registered [ 291.901427] LNet: 43921:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 291.901427] mask usage: [+|-] ... [ 291.959152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 291.967054] alg: No test for adler32 (adler32-zlib) [ 292.775722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 292.795881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 292.803785] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 293.103200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 293.110350] LNet: 44091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 293.112847] LNetError: 44091:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 293.121464] LNet: Removed LNI 192.168.204.29@tcp [ 293.129700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.151826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 293.160698] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 293.304889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 293.318513] LNet: Removed LNI 192.168.204.29@tcp [ 293.325752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.347225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 293.355928] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 293.493983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 293.510584] LNet: Removed LNI 192.168.204.29@tcp [ 293.517938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.540071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 293.549719] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 293.692517] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 293.726559] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 293.728556] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 293.730384] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 295.898450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.907586] LNet: 44229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 295.910279] LNet: 44229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 295.912782] LNetError: 44229:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 295.914845] LNetError: 44229:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 295.922403] LNet: Removed LNI 192.168.204.29@tcp [ 295.929588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.953310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 295.964930] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 298.041982] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 20:34:15 (1782866055) [ 298.091227] LNet: 44824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 298.093514] LNetError: 44824:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 298.103203] LNet: Removed LNI 192.168.204.29@tcp [ 298.304246] Key type .llcrypt unregistered [ 298.305722] Key type ._llcrypt unregistered [ 300.313745] Key type ._llcrypt registered [ 300.315527] Key type .llcrypt registered [ 300.330818] LNet: 45050:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 300.330818] mask usage: [+|-] ... [ 300.382393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 300.389279] alg: No test for adler32 (adler32-zlib) [ 301.186910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.204487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 301.213395] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 301.474415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.481793] LNet: 45223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.484194] LNetError: 45223:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 301.495308] LNet: Removed LNI 192.168.204.29@tcp [ 301.501986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.521954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 301.530361] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 303.633630] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 20:34:20 (1782866060) [ 303.663656] LNet: 45808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 303.666466] LNetError: 45808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 303.672649] LNet: Removed LNI 192.168.204.29@tcp [ 303.903282] Key type .llcrypt unregistered [ 303.905162] Key type ._llcrypt unregistered [ 305.702914] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 305.748950] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 305.751825] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 305.755040] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 307.953890] Key type ._llcrypt registered [ 307.955658] Key type .llcrypt registered [ 307.969348] LNet: 46091:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 307.969348] mask usage: [+|-] ... [ 308.024782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 308.030212] alg: No test for adler32 (adler32-zlib) [ 308.836399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.857894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 308.866446] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 309.153548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.161831] LNet: 46264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.163907] LNetError: 46264:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 309.171318] LNet: Removed LNI 192.168.204.29@tcp [ 309.177876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.199717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 309.207606] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 311.362361] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 20:34:28 (1782866068) [ 311.394541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 311.403167] LNet: 46857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.407298] LNetError: 46857:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 311.417431] LNet: Removed LNI 192.168.204.29@tcp [ 311.429033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.466341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 313.437265] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 20:34:30 (1782866070) [ 313.493642] LNet: 47442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.735437] Key type .llcrypt unregistered [ 313.737299] Key type ._llcrypt unregistered [ 315.728940] Key type ._llcrypt registered [ 315.729845] Key type .llcrypt registered [ 315.747542] LNet: 47667:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 315.747542] mask usage: [+|-] ... [ 315.800558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 315.807595] alg: No test for adler32 (adler32-zlib) [ 316.614312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.639971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 316.651480] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 316.965958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 316.972959] LNet: 47839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 316.975495] LNetError: 47839:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 316.988387] LNet: Removed LNI 192.168.204.29@tcp [ 316.995773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.017648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 317.026296] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 319.144914] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 20:34:36 (1782866076) [ 319.183765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.196503] LNet: 48427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.199995] LNetError: 48427:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 319.210424] LNet: Removed LNI 192.168.204.29@tcp [ 319.220202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.242572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 321.366213] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 20:34:38 (1782866078) [ 321.904127] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 322.471149] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 20:34:39 (1782866079) [ 323.008118] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 323.610709] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 20:34:40 (1782866080) [ 324.163191] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 324.655345] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 20:34:41 (1782866081) [ 325.120987] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 325.630753] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 20:34:42 (1782866082) [ 326.210944] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 326.780561] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 20:34:43 (1782866083) [ 326.819413] LNet: 50475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.025197] Key type .llcrypt unregistered [ 327.026901] Key type ._llcrypt unregistered [ 328.909791] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 328.970318] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 328.971906] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 328.973761] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 331.152330] Key type ._llcrypt registered [ 331.153415] Key type .llcrypt registered [ 331.167250] LNet: 50762:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 331.167250] mask usage: [+|-] ... [ 331.219047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 331.225952] alg: No test for adler32 (adler32-zlib) [ 332.027575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.052308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 332.059165] LNet: Added LNI 192.168.204.29@tcp [8/512/0/180] [ 332.061365] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 332.062673] LNet: 50800:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 332.094466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.100948] LNet: 50819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.103660] LNetError: 50819:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 332.105797] LNet: Removed LNI 10.1.2.3@tcp [ 332.106682] LNetError: 50819:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 332.114349] LNet: Removed LNI 192.168.204.29@tcp [ 332.121357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.143451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 332.151261] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 332.174101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.181043] LNet: 50844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.183309] LNetError: 50844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 332.194286] LNet: Removed LNI 192.168.204.29@tcp [ 332.201579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.223911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 332.232117] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 332.233847] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 332.234994] LNet: 50852:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 332.267084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.274875] LNet: 50870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.277795] LNetError: 50870:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 332.280973] LNet: Removed LNI 10.1.2.3@tcp [ 332.282694] LNetError: 50870:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 332.290299] LNet: Removed LNI 192.168.204.29@tcp [ 332.297307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.318361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 332.326774] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 332.328709] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 332.331265] LNet: 50878:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 332.392436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.401976] LNet: 50904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.404480] LNetError: 50904:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 332.407417] LNet: Removed LNI 10.1.2.3@tcp [ 332.408389] LNetError: 50904:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 332.419263] LNet: Removed LNI 192.168.204.29@tcp [ 332.425733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.446399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 332.456071] LNet: Added LNI 192.168.204.29@tcp [8/256/0/182] [ 332.458230] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 332.492436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.501757] LNet: 50932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.505467] LNetError: 50932:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 332.507931] LNet: Removed LNI 10.1.2.3@tcp [ 332.508824] LNetError: 50932:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 332.520321] LNet: Removed LNI 192.168.204.29@tcp [ 332.527820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.551354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 332.562133] LNet: Added LNI 192.168.204.29@tcp [8/256/0/182] [ 332.565394] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 332.566547] LNet: 50940:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 332.590407] LNet: Removed LNI 10.1.2.3@tcp [ 332.596281] LNet: Removed LNI 192.168.204.29@tcp [ 332.809256] Key type .llcrypt unregistered [ 332.810572] Key type ._llcrypt unregistered [ 336.385452] Key type ._llcrypt registered [ 336.387164] Key type .llcrypt registered [ 336.827879] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 20:34:53 (1782866093) [ 336.886746] LNet: 51752:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 336.886746] mask usage: [+|-] ... [ 336.932343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 336.936917] alg: No test for adler32 (adler32-zlib) [ 337.743743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.773588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 337.793818] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 338.075775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.29 [ 338.085097] LNet: 51915:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 338.089447] LNet: Removed LNI 192.168.204.29@tcp [ 339.992336] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 20:34:56 (1782866096) [ 340.021294] LNet: 52479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.023735] LNetError: 52479:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 340.029623] LNet: Removed LNI 192.168.204.29@tcp [ 340.249214] Key type .llcrypt unregistered [ 340.250867] Key type ._llcrypt unregistered [ 342.209541] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 342.299843] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 344.575421] Key type ._llcrypt registered [ 344.576459] Key type .llcrypt registered [ 344.590955] LNet: 52767:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 344.590955] mask usage: [+|-] ... [ 344.636395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 344.642375] alg: No test for adler32 (adler32-zlib) [ 345.579526] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 347.573080] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 20:35:04 (1782866104) [ 347.605494] LNet: 53390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 347.608467] LNetError: 53390:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 347.618546] LNet: Removed LNI 10.1.2.3@tcp [ 347.820549] Key type .llcrypt unregistered [ 347.821574] Key type ._llcrypt unregistered [ 349.733703] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 349.844175] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 349.846729] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 349.850246] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 352.116927] Key type ._llcrypt registered [ 352.118946] Key type .llcrypt registered [ 352.132343] LNet: 53670:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 352.132343] mask usage: [+|-] ... [ 352.179729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 352.185792] alg: No test for adler32 (adler32-zlib) [ 353.005610] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 355.118051] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 20:35:12 (1782866112) [ 355.609533] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 356.141305] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 20:35:13 (1782866113) [ 356.172447] LNet: 54575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.176553] LNetError: 54575:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 356.185844] LNet: Removed LNI 10.1.2.3@tcp [ 356.394371] Key type .llcrypt unregistered [ 356.395386] Key type ._llcrypt unregistered [ 358.449649] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 358.565085] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 358.567596] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 358.570272] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 360.835735] Key type ._llcrypt registered [ 360.836858] Key type .llcrypt registered [ 360.852783] LNet: 54854:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 360.852783] mask usage: [+|-] ... [ 360.904362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 360.909975] alg: No test for adler32 (adler32-zlib) [ 361.710575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.743159] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 364.451414] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 20:35:21 (1782866121) [ 364.585346] LNet: 55809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.587858] LNetError: 55809:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 364.600432] LNet: Removed LNI 10.1.2.3@tcp [ 364.806295] Key type .llcrypt unregistered [ 364.807437] Key type ._llcrypt unregistered [ 366.802256] Key type ._llcrypt registered [ 366.803975] Key type .llcrypt registered [ 366.819597] LNet: 56031:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 366.819597] mask usage: [+|-] ... [ 366.879727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 366.885176] alg: No test for adler32 (adler32-zlib) [ 367.684164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 367.692741] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 370.462909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 370.705613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_interrupt -r 1 [ 370.718687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_interrupt -r 1 [ 370.731706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 370.739304] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 370.784109] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 370.828794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 370.842406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 370.854065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 370.864879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 370.876440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 370.887321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 370.897729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 370.908849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 370.914935] LNet: 56244:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 370.921397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 370.931115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 370.941424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 370.951265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 370.963089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 370.974153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 370.985023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 371.195801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_dropped -r 1 [ 371.210251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_dropped -r 1 [ 371.225687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 371.320561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 371.333364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 371.344843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 371.355961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 371.366639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 371.376290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 371.387593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 371.398659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 371.410887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 371.422781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 371.434818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 371.446268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 371.457770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 371.468056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 371.478722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 371.673172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_aborted -r 1 [ 371.687743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_aborted -r 1 [ 371.701858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 371.709925] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 371.712013] LNet: Skipped 1 previous similar message [ 371.792283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 371.803430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 371.815434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 371.829742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 371.844128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 371.856953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 371.868870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 371.880120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 371.887432] LNet: 56463:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 371.889750] LNet: 56463:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 371.896723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 371.908216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 371.920512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 371.934673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 371.947507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 371.959979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 371.975756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 372.181315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_no_route -r 1 [ 372.200607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_no_route -r 1 [ 372.218207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 372.314947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 372.325032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 372.336448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 372.347702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 372.359585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 372.371726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 372.384348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 372.395968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 372.408646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 372.420394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 372.431380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 372.442955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 372.455986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 372.468069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 372.479340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 372.659212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_timeout -r 1 [ 372.675765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_timeout -r 1 [ 372.690977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 372.771946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 372.783590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 372.794689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 372.805737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 372.817278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 372.828567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 372.839451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 372.850271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 372.861274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 372.873858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 372.888276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 372.900339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 372.913312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 372.926374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 372.938787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 373.110343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_error -r 1 [ 373.126277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_error -r 1 [ 373.140262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 373.148095] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 373.151380] LNet: Skipped 3 previous similar messages [ 373.220056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 373.230553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 373.243244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 373.255677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 373.266987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 373.277617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 373.290209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 373.301786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 373.309407] LNet: 56790:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 373.311592] LNet: 56790:0:(api-ni.c:294:recovery_interval_set()) Skipped 2 previous similar messages [ 373.318105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 373.329614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 373.343197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 373.356860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 373.368883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 373.379470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 373.390515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 373.483499] LNet: 56831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.485941] LNetError: 56831:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 373.492296] LNet: Removed LNI 192.168.204.29@tcp [ 373.628421] Key type .llcrypt unregistered [ 373.630329] Key type ._llcrypt unregistered [ 378.674895] Key type ._llcrypt registered [ 378.675963] Key type .llcrypt registered [ 379.120030] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 20:35:36 (1782866136) [ 379.387224] Key type .llcrypt unregistered [ 379.389126] Key type ._llcrypt unregistered [ 381.139457] Key type ._llcrypt registered [ 381.140461] Key type .llcrypt registered [ 381.151233] LNet: 57879:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 381.151233] mask usage: [+|-] ... [ 381.199471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 381.204190] alg: No test for adler32 (adler32-zlib) [ 382.005877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 382.017183] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 384.705725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 387.672557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.684631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 387.691993] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 388.123281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_interrupt -r 1 [ 388.134848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_interrupt -r 1 [ 388.146834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_interrupt -r 1 [ 388.159178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_interrupt -r 1 [ 388.171502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_interrupt -r 1 [ 388.183838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_interrupt -r 1 [ 388.196626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_interrupt -r 1 [ 388.208828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_interrupt -r 1 [ 388.221794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 388.228624] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 388.231217] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 388.306233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 388.316246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 388.326211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 388.335943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 388.349261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 388.360187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 388.369822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 388.379163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 388.385576] LNet: 58306:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 388.391809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 388.401932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 388.412693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 388.424896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 388.436403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 388.452448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 388.467823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 388.630360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_dropped -r 1 [ 388.641938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_dropped -r 1 [ 388.654146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_dropped -r 1 [ 388.665359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_dropped -r 1 [ 388.677231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_dropped -r 1 [ 388.689292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_dropped -r 1 [ 388.701857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_dropped -r 1 [ 388.715484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_dropped -r 1 [ 388.732673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 388.742341] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 388.744562] LNet: Skipped 2 previous similar messages [ 388.813993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 388.824262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 388.834016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 388.845180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 388.856919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 388.872208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 388.882804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 388.892737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 388.900045] LNet: 58427:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 388.909415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 388.923458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 388.933377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 388.943857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 388.954421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 388.965183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 388.975929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 389.137842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_aborted -r 1 [ 389.151152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_aborted -r 1 [ 389.163491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_aborted -r 1 [ 389.176391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_aborted -r 1 [ 389.190396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_aborted -r 1 [ 389.204421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_aborted -r 1 [ 389.220119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_aborted -r 1 [ 389.232615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_aborted -r 1 [ 389.244595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 389.318765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 389.329951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 389.341022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 389.352198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 389.362925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 389.372094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 389.382210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 389.393134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 389.404291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 389.415931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 389.428633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 389.438931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 389.450702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 389.463727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 389.477607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 389.645989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_no_route -r 1 [ 389.658136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_no_route -r 1 [ 389.670082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_no_route -r 1 [ 389.682217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_no_route -r 1 [ 389.695413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_no_route -r 1 [ 389.707728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_no_route -r 1 [ 389.720506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_no_route -r 1 [ 389.733713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_no_route -r 1 [ 389.746683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 389.753601] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 389.755678] LNet: Skipped 6 previous similar messages [ 389.825691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 389.835253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 389.844960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 389.854525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 389.864461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 389.874907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 389.885521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 389.897629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 389.905612] LNet: 58670:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 389.907833] LNet: 58670:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 389.913486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 389.923460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 389.933191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 389.942968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 389.953538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 389.967649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 389.980934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 390.146572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_timeout -r 1 [ 390.158088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_timeout -r 1 [ 390.170188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_timeout -r 1 [ 390.182105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_timeout -r 1 [ 390.197459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_timeout -r 1 [ 390.212956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_timeout -r 1 [ 390.227678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_timeout -r 1 [ 390.239066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_timeout -r 1 [ 390.250712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 390.324682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 390.336928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 390.347228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 390.356981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 390.366761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 390.376967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 390.386599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 390.396505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 390.405574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 390.415823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 390.425268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 390.434934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 390.445284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 390.455517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 390.465618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 390.633528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_error -r 1 [ 390.647710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_error -r 1 [ 390.661159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_error -r 1 [ 390.674673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_error -r 1 [ 390.688492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_error -r 1 [ 390.700091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_error -r 1 [ 390.711655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_error -r 1 [ 390.723553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_error -r 1 [ 390.735799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 390.806608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 390.816368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 390.827562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 390.838612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 390.848573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 390.858873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 390.868991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 390.879697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 390.890765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 390.901089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 390.911178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 390.922900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 390.933988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 390.944852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 390.956515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 393.043114] LNet: 58987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.045479] LNetError: 58987:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 393.048400] LNet: Removed LNI 192.168.204.29@tcp [ 393.058276] LNet: Removed LNI 192.168.204.29@tcp1 [ 393.186620] Key type .llcrypt unregistered [ 393.188600] Key type ._llcrypt unregistered [ 398.261164] Key type ._llcrypt registered [ 398.262193] Key type .llcrypt registered [ 398.745748] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 20:35:55 (1782866155) [ 399.030355] Key type .llcrypt unregistered [ 399.031570] Key type ._llcrypt unregistered [ 400.998423] Key type ._llcrypt registered [ 400.999643] Key type .llcrypt registered [ 401.014429] LNet: 60020:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 401.014429] mask usage: [+|-] ... [ 401.070509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 401.075492] alg: No test for adler32 (adler32-zlib) [ 401.874573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 401.884661] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 404.623801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 404.819400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_dropped -r 1 [ 404.831133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_dropped -r 1 [ 404.842491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 404.849035] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 404.915772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 404.925911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 404.936151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 404.946707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 404.957413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 404.967597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 404.977236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 404.987472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 404.993532] LNet: 60234:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 404.999768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 405.009201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 405.019029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 405.028522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 405.038913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 405.048844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 405.058739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 405.230038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_error -r 1 [ 405.244380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_error -r 1 [ 405.257392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 405.332535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 405.342795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 405.352921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 405.363388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 405.373904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 405.384106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 405.393520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 405.403509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 405.414292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 405.424845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 405.435965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 405.447603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 405.458814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 405.470182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 405.480145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 405.651692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_timeout -r 1 [ 405.665210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_timeout -r 1 [ 405.679737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 405.689787] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 405.692183] LNet: Skipped 1 previous similar message [ 405.763754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 405.774500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 405.784276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 405.794577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 405.805783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 405.818547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 405.830912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 405.842466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 405.850137] LNet: 60452:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 405.854445] LNet: 60452:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 405.864634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 405.880692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 405.894111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 405.904593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 405.915375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 405.926740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 405.938302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 406.016601] LNet: 60493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 406.018823] LNetError: 60493:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 406.029284] LNet: Removed LNI 192.168.204.29@tcp [ 406.158290] Key type .llcrypt unregistered [ 406.159430] Key type ._llcrypt unregistered [ 411.254489] Key type ._llcrypt registered [ 411.255505] Key type .llcrypt registered [ 411.689333] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 20:36:08 (1782866168) [ 411.937433] Key type .llcrypt unregistered [ 411.938507] Key type ._llcrypt unregistered [ 413.783165] Key type ._llcrypt registered [ 413.784355] Key type .llcrypt registered [ 413.797895] LNet: 61527:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 413.797895] mask usage: [+|-] ... [ 413.856446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 413.861976] alg: No test for adler32 (adler32-zlib) [ 414.658645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 414.667740] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 417.408738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 420.006259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.016443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 420.022833] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 420.460702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_dropped -r 1 [ 420.475285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e remote_dropped -r 1 [ 420.488470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_dropped -r 1 [ 420.500564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e remote_dropped -r 1 [ 420.513747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e remote_dropped -r 1 [ 420.529659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e remote_dropped -r 1 [ 420.541557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e remote_dropped -r 1 [ 420.555851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e remote_dropped -r 1 [ 420.571114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 420.580840] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 420.582835] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.129@tcp [ 420.656515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 420.666952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 420.681725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 420.693434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 420.703890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 420.713867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 420.727797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 420.742834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 420.751512] LNet: 61955:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 420.759395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 420.769313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 420.782623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 420.795356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 420.808498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 420.823371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 420.834748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 420.845831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 421.010847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_error -r 1 [ 421.023263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e remote_error -r 1 [ 421.037754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_error -r 1 [ 421.050886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e remote_error -r 1 [ 421.063039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e remote_error -r 1 [ 421.076240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e remote_error -r 1 [ 421.091070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e remote_error -r 1 [ 421.110496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e remote_error -r 1 [ 421.123097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 421.129172] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 421.132514] LNet: Skipped 2 previous similar messages [ 421.203183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 421.213971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 421.224126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 421.234528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 421.244371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 421.254587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 421.264762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 421.274942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 421.280898] LNet: 62078:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 421.287769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 421.297815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 421.308210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 421.319264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 421.329908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 421.344460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 421.356405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 421.367778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 421.531930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_timeout -r 1 [ 421.545619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e remote_timeout -r 1 [ 421.560101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_timeout -r 1 [ 421.572317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e remote_timeout -r 1 [ 421.584124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e remote_timeout -r 1 [ 421.596610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e remote_timeout -r 1 [ 421.608476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e remote_timeout -r 1 [ 421.620624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e remote_timeout -r 1 [ 421.633903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 421.709739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 421.720184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 421.730482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 421.741698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 421.752290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 421.762493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 421.774841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 421.786304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 421.796848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 421.809348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 421.820637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 421.832197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 421.842584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 421.855824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 421.867191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 421.878622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 423.959082] LNet: 62278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 423.962510] LNetError: 62278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 423.965273] LNet: Removed LNI 192.168.204.29@tcp [ 423.974279] LNet: Removed LNI 192.168.204.29@tcp1 [ 424.113383] Key type .llcrypt unregistered [ 424.115466] Key type ._llcrypt unregistered [ 429.417032] Key type ._llcrypt registered [ 429.418068] Key type .llcrypt registered [ 429.847136] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 20:36:26 (1782866186) [ 430.339158] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 430.873820] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 20:36:27 (1782866187) [ 431.124181] Key type .llcrypt unregistered [ 431.125254] Key type ._llcrypt unregistered [ 433.117350] Key type ._llcrypt registered [ 433.118425] Key type .llcrypt registered [ 433.129336] LNet: 63604:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 433.129336] mask usage: [+|-] ... [ 433.178019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 433.182719] alg: No test for adler32 (adler32-zlib) [ 433.981989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 433.991462] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 436.918362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 437.126475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e network_timeout -r 1 [ 437.141099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e network_timeout -r 1 [ 437.152953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 437.159812] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 437.226781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 437.237429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 437.247450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 437.258273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 437.268358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 437.278938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 437.289824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 437.300683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 437.307558] LNet: 63818:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 437.314571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 437.325527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 437.336497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 437.346815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 437.356996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 437.368395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 437.452434] LNet: 63857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.455336] LNetError: 63857:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 437.468283] LNet: Removed LNI 192.168.204.29@tcp [ 437.614192] Key type .llcrypt unregistered [ 437.615256] Key type ._llcrypt unregistered [ 443.314525] Key type ._llcrypt registered [ 443.315609] Key type .llcrypt registered [ 443.327882] LNet: 64304:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 443.327882] mask usage: [+|-] ... [ 443.378246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 443.383370] alg: No test for adler32 (adler32-zlib) [ 444.185745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 444.196494] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 447.016991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 449.859148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.876744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 449.887599] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 450.398354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e network_timeout -r 1 [ 450.413537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e network_timeout -r 1 [ 450.428188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e network_timeout -r 1 [ 450.444667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e network_timeout -r 1 [ 450.457461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e network_timeout -r 1 [ 450.471670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e network_timeout -r 1 [ 450.486941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e network_timeout -r 1 [ 450.502632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e network_timeout -r 1 [ 450.517558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 450.526177] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 450.614333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 450.625648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 450.639534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 450.654183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 450.671928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 450.685155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 450.696967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 450.708083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 450.714499] LNet: 64731:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 450.720934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 450.731510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 450.741894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 450.754117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 450.767873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 450.783139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 452.860791] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 452.954376] LNet: 64805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 452.956754] LNetError: 64805:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 452.959305] LNet: Removed LNI 192.168.204.29@tcp [ 452.966548] LNet: Removed LNI 192.168.204.29@tcp1 [ 453.110643] Key type .llcrypt unregistered [ 453.111718] Key type ._llcrypt unregistered [ 458.563557] Key type ._llcrypt registered [ 458.564677] Key type .llcrypt registered [ 458.975475] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 20:36:55 (1782866215) [ 459.045536] LNet: 65603:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 459.045536] mask usage: [+|-] ... [ 459.094639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 459.099546] alg: No test for adler32 (adler32-zlib) [ 459.900627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.922091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 459.929722] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 460.164505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 460.171317] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 460.433776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 460.456123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 460.471541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 460.484647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 460.498179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 460.510945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 460.523115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.29@tcp [ 460.531644] LNetError: 65902:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.29@tcp to recovery [ 460.960237] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 460.960299] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 463.008303] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 463.673706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 467.104266] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 467.854311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 467.866047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 467.887298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.893730] LNet: 66020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 467.895957] LNetError: 66020:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 467.898394] LNet: Removed LNI 192.168.204.29@tcp [ 467.905257] LNet: Removed LNI 192.168.204.29@tcp1 [ 467.911017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.927286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 467.938969] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 468.181563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 468.188885] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 468.460160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 468.472332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 468.492579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 468.507244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 468.519810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 468.533668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 468.545374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 468.556887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.29@tcp [ 468.563763] LNetError: 66292:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.29@tcp to recovery [ 471.008433] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 471.012457] LNet: Skipped 1 previous similar message [ 471.770159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 475.104392] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 480.184145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 480.196105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 480.207777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 482.432790] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 20:37:19 (1782866239) [ 482.464440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.475685] LNet: 67015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 482.478264] LNetError: 67015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 482.480425] LNetError: 67015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 482.483310] LNet: Removed LNI 192.168.204.29@tcp [ 482.491744] LNet: Removed LNI 192.168.204.29@tcp1 [ 482.500067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.517465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 482.525550] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 482.749962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 482.757144] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 483.025698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 483.053067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 483.064815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 483.079177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 483.093838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 483.106739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 483.121199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.29@tcp --health 0 [ 483.159723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 483.552101] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.29@tcp [ 484.576405] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 484.580204] LNet: Skipped 1 previous similar message [ 499.600927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 499.652746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 499.665387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 499.677236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.29@tcp --health 500 [ 499.712337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 506.956891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 506.979666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.989059] LNet: 67534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 506.991942] LNetError: 67534:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 506.994854] LNetError: 67534:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 506.998236] LNet: Removed LNI 192.168.204.29@tcp [ 507.008299] LNet: Removed LNI 192.168.204.29@tcp1 [ 507.015134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.034823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 507.045037] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 507.304552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 507.313155] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 507.582643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 507.594990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 507.616767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 507.628296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 507.639423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 507.653550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 507.669574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 507.684156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.29@tcp --health 0 [ 508.064298] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 508.066715] LNet: Skipped 3 previous similar messages [ 508.788578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 519.256813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 519.268598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 519.282420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 521.720358] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 20:37:58 (1782866278) [ 521.885225] LNet: 68590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.887386] LNetError: 68590:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 521.889424] LNetError: 68590:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 521.892631] LNet: Removed LNI 192.168.204.29@tcp [ 521.899577] LNet: Removed LNI 192.168.204.29@tcp1 [ 522.142569] Key type .llcrypt unregistered [ 522.143637] Key type ._llcrypt unregistered [ 524.126106] Key type ._llcrypt registered [ 524.127266] Key type .llcrypt registered [ 524.138718] LNet: 68812:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 524.138718] mask usage: [+|-] ... [ 524.187472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 524.192426] alg: No test for adler32 (adler32-zlib) [ 524.990134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 524.999239] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 528.324859] Lustre: DEBUG MARKER: Initial discovery [ 528.360159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.129@tcp [ 529.111593] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 529.129107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 529.145418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.129@tcp [ 529.154621] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 529.163112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 529.179247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.129@tcp -r 1 -m GET -l 3 [ 529.195135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.129@tcp -r 1 -m GET -e local_error [ 529.208242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.129@tcp -r 1 -m PUT -l 6 [ 529.677869] Lustre: DEBUG MARKER: Force oleg429-client.virtnet to discover 192.168.204.129@tcp (in background) [ 529.711786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.129@tcp [ 530.145145] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 530.160520] Lustre: DEBUG MARKER: Force oleg429-server to discover 192.168.204.29@tcp [ 530.929280] Lustre: DEBUG MARKER: Wait for 69345 [ 532.768199] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 533.226062] Lustre: DEBUG MARKER: Finished wait on 69345 [ 533.254550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 533.270518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 533.362310] LNet: 69798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 533.364611] LNetError: 69798:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 533.373282] LNet: Removed LNI 192.168.204.29@tcp [ 533.510221] Key type .llcrypt unregistered [ 533.511310] Key type ._llcrypt unregistered [ 538.669768] Key type ._llcrypt registered [ 538.670897] Key type .llcrypt registered [ 539.101918] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 20:38:16 (1782866296) [ 539.248341] Key type .llcrypt unregistered [ 539.249884] Key type ._llcrypt unregistered [ 541.161786] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 541.204510] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 541.206041] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 541.207430] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 543.473610] Key type ._llcrypt registered [ 543.475117] Key type .llcrypt registered [ 543.491258] LNet: 70840:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 543.491258] mask usage: [+|-] ... [ 543.548708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 543.554196] alg: No test for adler32 (adler32-zlib) [ 544.354910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.373436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 544.381786] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 544.606739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 544.613799] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 547.501825] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 20:38:24 (1782866304) [ 547.533706] LNet: 71866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 547.536442] LNetError: 71866:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 547.538668] LNet: Removed LNI 10.1.2.3@tcp [ 547.546510] LNet: Removed LNI 192.168.204.29@tcp [ 547.761267] Key type .llcrypt unregistered [ 547.762963] Key type ._llcrypt unregistered [ 549.755678] Key type ._llcrypt registered [ 549.757624] Key type .llcrypt registered [ 549.770663] LNet: 72121:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 549.770663] mask usage: [+|-] ... [ 549.819451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 549.825511] alg: No test for adler32 (adler32-zlib) [ 550.628176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.654158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 550.663940] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 550.900461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 550.910552] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 551.210599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.29@tcp [ 552.224124] LNetError: Unexpected error -22 connecting to 192.168.204.29@tcp at host 192.168.204.29:988 [ 553.317196] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 20:38:30 (1782866310) [ 553.348858] LNet: 72995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 553.351542] LNetError: 72995:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 553.355120] LNet: Removed LNI 10.1.2.3@tcp [ 553.362874] LNet: Removed LNI 192.168.204.29@tcp [ 553.557333] Key type .llcrypt unregistered [ 553.558369] Key type ._llcrypt unregistered [ 555.453994] Key type ._llcrypt registered [ 555.456061] Key type .llcrypt registered [ 555.471341] LNet: 73233:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 555.471341] mask usage: [+|-] ... [ 555.522568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 555.527355] alg: No test for adler32 (adler32-zlib) [ 556.339224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.360315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 556.370170] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 556.605634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 556.612259] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 556.897877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.29@tcp1 --nid 192.168.204.29@tcp2 [ 558.270683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp1 192.168.204.29@tcp2 [ 558.281652] LNetError: 73836:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.29@tcp1 to 192.168.204.29@tcp2 - Route table may be misconfigured [ 558.291827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp2 192.168.204.29@tcp1 [ 560.479668] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 20:38:37 (1782866317) [ 560.516182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.524804] LNet: 74406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 560.527211] LNetError: 74406:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 560.529539] LNet: Removed LNI 192.168.204.29@tcp1 [ 561.574601] LNet: Removed LNI 192.168.204.29@tcp2 [ 561.582957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.602729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 561.613208] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 561.895760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 561.903511] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 562.221532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 562.240718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.29@tcp [ 562.258493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp1 192.168.204.29@tcp1 [ 562.276487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.29@tcp -d 192.168.204.29@tcp -e network_timeout [ 562.292650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -e network_timeout [ 562.308418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -e network_timeout [ 562.324786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -e network_timeout [ 562.340244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 562.349782] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 562.357746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 562.403510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 562.444387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 562.657079] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 564.532857] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 20:38:41 (1782866321) [ 564.561874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.571804] LNet: 75264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 564.576236] LNetError: 75264:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 564.580662] LNetError: 75264:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 564.584583] LNet: Removed LNI 192.168.204.29@tcp [ 565.606566] LNet: Removed LNI 192.168.204.29@tcp1 [ 565.616116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.646366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 565.736038] LNet: 75307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 565.864262] Key type .llcrypt unregistered [ 565.865323] Key type ._llcrypt unregistered [ 569.365664] Key type ._llcrypt registered [ 569.366841] Key type .llcrypt registered [ 569.929637] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 20:38:46 (1782866326) [ 570.003409] LNet: 76076:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 570.003409] mask usage: [+|-] ... [ 570.057190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 570.063599] alg: No test for adler32 (adler32-zlib) [ 570.867250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.437588] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 572.010978] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 20:38:48 (1782866328) [ 572.044444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.052834] LNet: 76401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 572.062185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.078664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 572.085875] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 572.291709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 572.298382] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 572.558877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 572.575629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp1 [ 572.591737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp1 [ 575.093493] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 20:38:51 (1782866331) [ 575.954958] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg429-server [ 576.735665] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 20:38:53 (1782866333) [ 577.427290] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg429-server [ 577.968704] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 20:38:54 (1782866334) [ 578.602838] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg429-server [ 579.257688] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 20:38:56 (1782866336) [ 580.012548] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg429-server [ 580.615186] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 20:38:57 (1782866337) [ 581.318878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg429-server [ 581.951497] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 20:38:58 (1782866338) [ 582.574876] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg429-server [ 583.122507] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 20:39:00 (1782866340) [ 583.700510] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg429-server [ 584.225021] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 20:39:01 (1782866341) [ 584.777505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg429-server [ 585.353038] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 20:39:02 (1782866342) [ 586.036096] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg429-server [ 586.578798] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 20:39:03 (1782866343) [ 586.617990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.627377] LNet: 80274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 586.629808] LNetError: 80274:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 586.634598] LNet: Removed LNI 192.168.204.29@tcp [ 587.686749] LNet: Removed LNI 192.168.204.29@tcp1 [ 587.698484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.726279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 587.737184] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 588.032929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 588.064887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 588.120646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.127337] LNet: 80423:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.129371] LNetError: 80423:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 588.131444] LNetError: 80423:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 588.138267] LNet: Removed LNI 192.168.204.29@tcp [ 588.145554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.164608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 588.173193] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 588.403572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 588.462715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.478309] LNet: Removed LNI 192.168.204.29@tcp [ 589.485412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.507071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 589.515877] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 589.746828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 589.776572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 589.835335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.843485] LNetError: 80716:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 589.845729] LNetError: 80716:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 589.853394] LNet: Removed LNI 192.168.204.29@tcp [ 589.860934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.881939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 589.892143] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 590.141804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 590.210823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.219287] LNet: 80861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.222115] LNet: 80861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 591.270322] LNet: Removed LNI 192.168.204.29@tcp [ 591.277902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.296311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 591.306141] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 591.542493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 591.568265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 591.629723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.678301] LNet: Removed LNI 192.168.204.29@tcp [ 592.684783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.702022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 592.712176] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 592.968585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 594.067664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.074504] LNetError: 81162:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.076366] LNetError: 81162:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 594.084345] LNet: Removed LNI 192.168.204.29@tcp [ 594.090072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.105205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 594.112329] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 594.343628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 594.368799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 594.430972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.437433] LNet: 81309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 594.439816] LNet: 81309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 595.494269] LNet: Removed LNI 192.168.204.29@tcp [ 595.501637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.518323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 595.525882] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 595.763406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 595.840390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.902299] LNet: Removed LNI 192.168.204.29@tcp [ 596.909704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.926327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 596.934991] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 597.182939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 597.208829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 597.282103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.310304] LNet: Removed LNI 192.168.204.29@tcp [ 598.317883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.338805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 598.349187] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 598.630445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 598.714742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.721444] LNetError: 81747:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 598.723527] LNetError: 81747:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 598.733496] LNet: Removed LNI 192.168.204.29@tcp [ 598.741558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.767924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 598.778311] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 599.047049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 599.082429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 600.220303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.254325] LNet: Removed LNI 192.168.204.29@tcp [ 601.263021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.282412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 601.290886] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 601.528282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 601.605699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.662295] LNet: Removed LNI 192.168.204.29@tcp [ 602.669490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.688602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 602.696931] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 602.933044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 602.956895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.965092] LNet: 82180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.967208] LNet: 82180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 602.976291] LNet: Removed LNI 192.168.204.29@tcp [ 602.982238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.001505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 603.010241] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 603.247974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 605.530197] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 20:39:22 (1782866362) [ 605.590833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.607104] LNet: Removed LNI 192.168.204.29@tcp [ 605.614458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.634460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 605.642800] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/180] [ 606.068785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.082410] LNet: Removed LNI 192.168.204.29@tcp231 [ 606.091974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.119631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 606.130895] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/180] [ 606.134929] LNet: 83030:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 606.268637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 606.284555] LNet: Removed LNI 192.168.204.29@tcp231 [ 606.291437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 606.301422] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/180] [ 606.700751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.714306] LNet: Removed LNI 192.168.204.29@tcp231 [ 606.721257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.744535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 606.753886] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/0] [ 606.756039] LNet: 83182:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 606.888431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 606.906515] LNet: Removed LNI 192.168.204.29@tcp231 [ 606.913304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 606.924724] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/0] [ 607.293908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.300708] LNetError: 83325:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 607.302887] LNetError: 83325:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 607.310268] LNet: Removed LNI 192.168.204.29@tcp231 [ 607.317044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.338801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 607.347820] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/60] [ 607.481097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 607.502547] LNet: Removed LNI 192.168.204.29@tcp231 [ 607.510297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 607.523794] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/60] [ 607.897166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.910295] LNet: Removed LNI 192.168.204.29@tcp231 [ 607.916060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.941968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 607.949836] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/180] [ 607.952250] LNet: 83484:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 607.954389] LNet: 83484:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 608.073170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 608.090366] LNet: Removed LNI 192.168.204.29@tcp231 [ 608.095718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 608.103594] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/180] [ 610.501625] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 20:39:27 (1782866367) [ 610.528923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.543715] LNet: Removed LNI 192.168.204.29@tcp231 [ 610.551393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.574861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 610.582418] LNet: Added LNI 192.168.204.29@tcp232 [8/256/0/180] [ 610.810966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 610.825749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 610.844342] LNet: Removed LNI 192.168.204.29@tcp232 [ 610.849970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 610.858699] LNet: Added LNI 192.168.204.29@tcp232 [8/256/0/180] [ 611.111934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 611.126299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 611.139595] LNet: Removed LNI 192.168.204.29@tcp232 [ 611.147845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 611.156487] LNet: Added LNI 192.168.204.29@tcp232 [8/256/0/180] [ 611.388544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 611.400680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 611.412407] LNet: Removed LNI 192.168.204.29@tcp232 [ 611.417818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 611.427630] LNet: Added LNI 192.168.204.29@tcp232 [8/256/0/180] [ 611.653318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 611.667721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 611.684417] LNet: Removed LNI 192.168.204.29@tcp232 [ 611.689663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 613.653233] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 20:39:30 (1782866370) [ 614.007380] Key type .llcrypt unregistered [ 614.008941] Key type ._llcrypt unregistered [ 616.035324] Key type ._llcrypt registered [ 616.036532] Key type .llcrypt registered [ 616.049638] LNet: 85548:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 616.049638] mask usage: [+|-] ... [ 616.103136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 616.108255] alg: No test for adler32 (adler32-zlib) [ 616.908735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 616.919407] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 619.824211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 622.485650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.500059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 622.509630] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 622.979641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_interrupt -r 2 [ 622.993033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_interrupt -r 2 [ 623.005267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_interrupt -r 2 [ 623.017502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_interrupt -r 2 [ 623.030111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_interrupt -r 2 [ 623.042951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_interrupt -r 2 [ 623.055908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_interrupt -r 2 [ 623.070760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_interrupt -r 2 [ 623.084643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 623.093520] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 623.096369] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 623.188327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 623.199744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 623.209758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 623.219294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 623.230655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 623.242246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 623.252327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 623.265260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 623.272476] LNet: 85979:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 623.279829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 623.291277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 623.302634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 623.314106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 623.327161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 623.339522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 623.514888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_dropped -r 2 [ 623.527249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_dropped -r 2 [ 623.542812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_dropped -r 2 [ 623.559164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_dropped -r 2 [ 623.573892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_dropped -r 2 [ 623.588635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_dropped -r 2 [ 623.604764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_dropped -r 2 [ 623.618288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_dropped -r 2 [ 623.632713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 623.643418] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 623.730901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 623.741383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 623.752112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 623.763425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 623.776556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 623.788972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 623.799648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 623.810267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 623.816636] LNet: 86099:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 623.824498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 623.838509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 623.852298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 623.866826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 623.882631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 623.897923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 624.123697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_aborted -r 2 [ 624.137785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_aborted -r 2 [ 624.152279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_aborted -r 2 [ 624.165777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_aborted -r 2 [ 624.178443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_aborted -r 2 [ 624.190715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_aborted -r 2 [ 624.203282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_aborted -r 2 [ 624.215185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_aborted -r 2 [ 624.228225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 624.318806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 624.329390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 624.339401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 624.351268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 624.365868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 624.379467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 624.391498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 624.403510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 624.413867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 624.424706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 624.434877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 624.445747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 624.457442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 624.472475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 624.659305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_no_route -r 2 [ 624.678671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_no_route -r 2 [ 624.691813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_no_route -r 2 [ 624.704836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_no_route -r 2 [ 624.717850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_no_route -r 2 [ 624.729984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_no_route -r 2 [ 624.746418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_no_route -r 2 [ 624.758763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_no_route -r 2 [ 624.772720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 624.781354] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 624.783478] LNet: Skipped 1 previous similar message [ 624.858703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 624.868269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 624.877586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 624.888054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 624.897909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 624.908508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 624.920944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 624.933594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 624.941467] LNet: 86340:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 624.944008] LNet: 86340:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 624.950701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 624.963178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 624.975411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 624.987233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 624.998660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 625.010764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 625.193450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_timeout -r 2 [ 625.207440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_timeout -r 2 [ 625.221628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_timeout -r 2 [ 625.238189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_timeout -r 2 [ 625.255733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_timeout -r 2 [ 625.272177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_timeout -r 2 [ 625.286695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_timeout -r 2 [ 625.303413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_timeout -r 2 [ 625.319913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 625.410626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 625.421642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 625.431990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 625.442242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 625.452633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 625.463471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 625.475374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 625.487249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 625.498420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 625.509717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 625.522506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 625.534070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 625.546650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 625.559830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 625.730081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_dropped -r 2 [ 625.742099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e remote_dropped -r 2 [ 625.753487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_dropped -r 2 [ 625.766533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e remote_dropped -r 2 [ 625.782346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e remote_dropped -r 2 [ 625.801977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e remote_dropped -r 2 [ 625.822377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e remote_dropped -r 2 [ 625.835502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e remote_dropped -r 2 [ 625.848122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 625.856279] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.129@tcp [ 625.925493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 625.937547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 625.949870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 625.960966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 625.971893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 625.982425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 625.993592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 626.005395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 626.016381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 626.027783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 626.039345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 626.050612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 626.061281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 626.073265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 628.183719] LNet: 86653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.187276] LNetError: 86653:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 628.190492] LNet: Removed LNI 192.168.204.29@tcp [ 628.199284] LNet: Removed LNI 192.168.204.29@tcp1 [ 628.333448] Key type .llcrypt unregistered [ 628.334545] Key type ._llcrypt unregistered [ 633.225871] Key type ._llcrypt registered [ 633.226932] Key type .llcrypt registered [ 633.642453] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 20:39:50 (1782866390) [ 633.710968] LNet: 87457:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 633.710968] mask usage: [+|-] ... [ 633.757607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 633.762719] alg: No test for adler32 (adler32-zlib) [ 634.563683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.584628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 634.592788] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 634.871277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 634.882233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.29@tcp [ 634.889836] LNetError: 87629:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.29@tcp to recovery [ 634.919727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.29@tcp [ 634.951458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 636.871619] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 20:39:53 (1782866393) [ 636.905183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.914280] LNet: 88214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.916689] LNetError: 88214:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.922656] LNet: Removed LNI 192.168.204.29@tcp [ 636.931512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.950419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 636.958826] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 637.187448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 637.195803] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 637.494769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.29@tcp [ 637.502600] LNetError: 88484:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.29@tcp to recovery [ 637.984330] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 637.984976] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 638.576982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.29@tcp [ 641.619659] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 20:39:58 (1782866398) [ 642.211199] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg429-server [ 642.793065] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 20:39:59 (1782866399) [ 643.419085] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg429-server [ 643.975150] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 20:40:00 (1782866400) [ 644.526056] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg429-server [ 645.030583] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 20:40:02 (1782866402) [ 645.073706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.083136] LNet: 90106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.085914] LNetError: 90106:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.088408] LNet: Removed LNI 192.168.204.29@tcp [ 645.096153] LNet: Removed LNI 192.168.204.29@tcp1 [ 645.103324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.121411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 645.129228] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 645.351870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 645.372788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 645.387505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 645.410936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 645.427868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 645.441228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 647.409503] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 20:40:04 (1782866404) [ 647.454071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.462905] LNet: 90830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.466951] LNetError: 90830:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.470708] LNetError: 90830:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 647.479560] LNet: Removed LNI 192.168.204.29@tcp [ 647.487670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.505719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 647.513653] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 650.001161] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 20:40:06 (1782866406) [ 650.048896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.063524] LNet: 91528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.068314] LNetError: 91528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 650.077872] LNet: Removed LNI 192.168.204.29@tcp [ 650.091758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.119934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 650.131338] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 652.839300] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 20:40:09 (1782866409) [ 653.493315] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 654.191881] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 20:40:11 (1782866411) [ 654.413786] LNet: 92567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.418487] LNetError: 92567:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.431490] LNet: Removed LNI 192.168.204.29@tcp [ 654.774284] Key type .llcrypt unregistered [ 654.776522] Key type ._llcrypt unregistered [ 656.592755] Key type ._llcrypt registered [ 656.593783] Key type .llcrypt registered [ 656.605709] LNet: 92796:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 656.605709] mask usage: [+|-] ... [ 656.651026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 656.655632] alg: No test for adler32 (adler32-zlib) [ 657.450427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 657.458551] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 660.565340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 662.255309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.129@tcp [ 662.351597] LNet: 92983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.353986] LNetError: 92983:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 662.365343] LNet: Removed LNI 192.168.204.29@tcp [ 662.499371] Key type .llcrypt unregistered [ 662.501317] Key type ._llcrypt unregistered [ 666.267918] Key type ._llcrypt registered [ 666.268880] Key type .llcrypt registered [ 666.677398] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 667.194784] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 667.734132] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 20:40:24 (1782866424) [ 668.238081] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 668.826989] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 20:40:25 (1782866425) [ 669.461089] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg429-server [ 670.022043] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 20:40:27 (1782866427) [ 670.655414] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg429-server [ 671.203372] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 20:40:28 (1782866428) [ 671.590347] LNet: 95024:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 671.590347] mask usage: [+|-] ... [ 671.727231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 671.768526] alg: No test for adler32 (adler32-zlib) [ 672.623299] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 672.711577] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 674.296103] Key type lgssc registered [ 674.673859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 682.449353] Key type lgssc unregistered [ 682.571380] LNet: 96153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.573946] LNetError: 96153:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 682.585316] LNet: Removed LNI 192.168.204.29@tcp [ 682.834270] Key type .llcrypt unregistered [ 682.835318] Key type ._llcrypt unregistered [ 687.739524] Key type ._llcrypt registered [ 687.741199] Key type .llcrypt registered [ 688.163043] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 20:40:45 (1782866445) [ 688.228710] LNet: 96947:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 688.228710] mask usage: [+|-] ... [ 688.274432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 688.279207] alg: No test for adler32 (adler32-zlib) [ 689.079383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.101945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 689.117298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 689.132775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 689.147851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 691.138805] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 20:40:48 (1782866448) [ 691.174806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.184034] LNet: 97561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.196690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.215629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 691.226492] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 691.529801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.539291] LNetError: 97708:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.547289] LNet: Removed LNI 192.168.204.29@tcp [ 691.553085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.569151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 691.577377] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 691.711321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.719243] LNet: 97736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.722328] LNet: 97736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 691.732260] LNet: Removed LNI 192.168.204.29@tcp [ 691.738495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.756211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 691.765615] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 694.175851] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 20:40:51 (1782866451) [ 694.245764] LNet: 98320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.250900] LNetError: 98320:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 694.255956] LNetError: 98320:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 694.267070] LNet: Removed LNI 192.168.204.29@tcp [ 694.595331] Key type .llcrypt unregistered [ 694.597443] Key type ._llcrypt unregistered [ 696.622258] Key type ._llcrypt registered [ 696.623271] Key type .llcrypt registered [ 696.637454] LNet: 98556:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 696.637454] mask usage: [+|-] ... [ 696.683484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 696.688294] alg: No test for adler32 (adler32-zlib) [ 697.490601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.520971] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 697.523465] LNet: *** cfs_fail_loc=e003, val=100*** [ 697.770787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 697.778906] Ping reply received with mapping: [ 697.780241] found metadata reply NI[192.168.204.29@tcp] [ 697.781581] 1d cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 697.783049] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 697.784414] 5a 5a 5a 5a 5a 5a 5a 5a [ 697.835117] LNet: 98722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.837957] LNetError: 98722:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 697.846319] LNet: Removed LNI 192.168.204.29@tcp [ 698.060187] Key type .llcrypt unregistered [ 698.061278] Key type ._llcrypt unregistered [ 701.951439] Key type ._llcrypt registered [ 701.952835] Key type .llcrypt registered [ 702.501719] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 20:40:59 (1782866459) [ 702.696178] Key type .llcrypt unregistered [ 702.697492] Key type ._llcrypt unregistered [ 705.120421] Key type ._llcrypt registered [ 705.121511] Key type .llcrypt registered [ 705.132284] LNet: 99716:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 705.132284] mask usage: [+|-] ... [ 705.176779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 705.181712] alg: No test for adler32 (adler32-zlib) [ 706.056445] LNetError: 99766:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 706.198212] Key type .llcrypt unregistered [ 706.199811] Key type ._llcrypt unregistered [ 708.482026] Key type ._llcrypt registered [ 708.483141] Key type .llcrypt registered [ 708.999966] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 20:41:05 (1782866465) [ 709.323789] Key type .llcrypt unregistered [ 709.325675] Key type ._llcrypt unregistered [ 711.551879] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 711.554832] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 713.789114] Key type ._llcrypt registered [ 713.791058] Key type .llcrypt registered [ 713.810880] LNet: 100633:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 713.810880] mask usage: [+|-] ... [ 713.885272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 713.893154] alg: No test for adler32 (adler32-zlib) [ 714.699693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.721948] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 719.112509] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 20:41:16 (1782866476) [ 719.271693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.288552] LNet: 101544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.294522] LNetError: 101544:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 719.311542] LNet: Removed LNI 10.1.2.3@tcp [ 719.324200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.735566] Key type .llcrypt unregistered [ 719.737907] Key type ._llcrypt unregistered [ 722.172032] Key type ._llcrypt registered [ 722.173117] Key type .llcrypt registered [ 722.183895] LNet: 101775:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 722.183895] mask usage: [+|-] ... [ 722.228496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 722.233567] alg: No test for adler32 (adler32-zlib) [ 723.040939] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 723.065512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 723.127977] LNet: 101832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.132174] LNetError: 101832:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 723.142433] LNet: Removed LNI 192.168.204.29@tcp [ 723.369303] Key type .llcrypt unregistered [ 723.370580] Key type ._llcrypt unregistered [ 727.823388] Key type ._llcrypt registered [ 727.824803] Key type .llcrypt registered [ 728.264854] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 20:41:25 (1782866485) [ 728.449296] Key type .llcrypt unregistered [ 728.450435] Key type ._llcrypt unregistered [ 730.372160] Key type ._llcrypt registered [ 730.373414] Key type .llcrypt registered [ 730.384833] LNet: 102814:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 730.384833] mask usage: [+|-] ... [ 730.430726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 730.436537] alg: No test for adler32 (adler32-zlib) [ 733.273523] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 20:41:30 (1782866490) [ 733.317514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 733.344484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.385587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 733.402717] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 733.688416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 733.699929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 733.715603] LNet: Removed LNI 192.168.204.29@tcp [ 733.796595] LNet: 103573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.931195] Key type .llcrypt unregistered [ 733.932232] Key type ._llcrypt unregistered [ 738.020702] Key type ._llcrypt registered [ 738.021639] Key type .llcrypt registered [ 738.703119] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 20:41:35 (1782866495) [ 738.820895] LNet: 104336:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 738.820895] mask usage: [+|-] ... [ 738.881882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 738.887540] alg: No test for adler32 (adler32-zlib) [ 739.730350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.761762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 739.773351] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 740.173805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 740.203652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.29@tcp [ 740.218127] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 740.221468] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 740.224410] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 740, 0, 0 [ 740.228631] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 740.231151] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 740.234375] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 740.237371] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 740, 0, 0 [ 740.241428] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 740.243645] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 740.246615] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 740.249407] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 740, 0, 790 [ 740.253052] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 740.255122] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 740.257868] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 740.260283] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 740, 0, 790 [ 740.263667] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 742.869710] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 20:41:39 (1782866499) [ 743.131250] LNet: 105110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.137139] LNetError: 105110:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.150097] LNet: Removed LNI 192.168.204.29@tcp [ 743.492479] Key type .llcrypt unregistered [ 743.494641] Key type ._llcrypt unregistered [ 745.625472] Key type ._llcrypt registered [ 745.626472] Key type .llcrypt registered [ 745.640377] LNet: 105333:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 745.640377] mask usage: [+|-] ... [ 745.684711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 745.689657] alg: No test for adler32 (adler32-zlib) [ 746.498437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 746.509087] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 749.554268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 752.376279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.391837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 752.402309] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 752.767511] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 752.839858] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 752.846092] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 752.848202] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 755.032980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 755.043133] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 758.554502] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 20:41:55 (1782866515) [ 758.585166] LNet: 106607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.588336] LNetError: 106607:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.591304] LNet: Removed LNI 192.168.204.29@tcp [ 759.648126] LNet: Removed LNI 192.168.204.29@tcp1 [ 759.649389] LNetError: 106607:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 759.651974] LNetError: 106607:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 759.660678] LNet: Removed LNI 10.1.2.3@tcp99 [ 759.885224] Key type .llcrypt unregistered [ 759.886350] Key type ._llcrypt unregistered [ 761.795041] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 761.796632] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 763.960337] Key type ._llcrypt registered [ 763.961601] Key type .llcrypt registered [ 763.973260] LNet: 106878:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 763.973260] mask usage: [+|-] ... [ 764.022568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 764.027611] alg: No test for adler32 (adler32-zlib) [ 764.825598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.841463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 764.848815] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 765.072626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 765.080890] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 765.730906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 765.747177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 765.762630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 765.769578] LNetError: 107357:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 765.780952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 765.796089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 765.933800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 765.958950] LNet: 107377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.961900] LNetError: 107377:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 765.964643] LNet: Removed LNI 10.1.2.3@tcp [ 767.014462] LNet: Removed LNI 192.168.204.29@tcp [ 767.280306] Key type .llcrypt unregistered [ 767.282202] Key type ._llcrypt unregistered [ 771.116117] Key type ._llcrypt registered [ 771.117277] Key type .llcrypt registered [ 771.538147] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 20:42:08 (1782866528) [ 771.607118] LNet: 108174:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 771.607118] mask usage: [+|-] ... [ 771.654141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 771.659807] alg: No test for adler32 (adler32-zlib) [ 772.457661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.474357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 772.482316] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 772.709405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 772.735176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg429-client@tcp [ 774.810868] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 20:42:11 (1782866531) [ 774.845176] LNet: 108901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.847852] LNetError: 108901:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.854703] LNet: Removed LNI 192.168.204.29@tcp [ 775.061233] Key type .llcrypt unregistered [ 775.062404] Key type ._llcrypt unregistered [ 777.047910] Key type ._llcrypt registered [ 777.048965] Key type .llcrypt registered [ 777.100236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 777.105436] alg: No test for adler32 (adler32-zlib) [ 777.901523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 777.910799] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 777.922781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 779.945871] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 20:42:16 (1782866536) [ 780.433535] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 780.966645] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 20:42:17 (1782866537) [ 781.460719] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 782.031412] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 20:42:19 (1782866539) [ 782.061945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 782.071112] LNet: 110319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.073489] LNetError: 110319:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.083993] LNet: Removed LNI 192.168.204.29@tcp [ 782.093356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.112496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 782.120465] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 782.366858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 782.384404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 784.430252] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 20:42:21 (1782866541) [ 784.463951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.472577] LNet: 111028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.475259] LNetError: 111028:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.490877] LNet: Removed LNI 192.168.204.29@tcp [ 784.498525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.517073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 784.530243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 784.758268] Key type .llcrypt unregistered [ 784.759530] Key type ._llcrypt unregistered [ 788.360376] Key type ._llcrypt registered [ 788.361500] Key type .llcrypt registered [ 788.800489] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 20:42:25 (1782866545) [ 788.862502] LNet: 111837:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 788.862502] mask usage: [+|-] ... [ 788.911816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 788.916865] alg: No test for adler32 (adler32-zlib) [ 789.718271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.739898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 789.748321] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 789.979986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 789.998678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 791.993360] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 20:42:29 (1782866549) [ 792.022294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.030819] LNet: 112580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.033311] LNetError: 112580:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.041588] LNet: Removed LNI 192.168.204.29@tcp [ 792.049728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.072290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 792.083365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 794.027850] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 20:42:31 (1782866551) [ 794.100874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.108512] LNet: 113165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.116525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.137429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 794.262738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 794.274826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 794.295381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 794.431498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 794.442291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 794.466993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.479185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.499936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 794.632484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 794.650514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 794.674099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 794.804857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 794.822133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 797.032898] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 20:42:33 (1782866553) [ 797.089458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.103507] LNet: 113797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.105909] LNet: 113797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 797.115188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.138358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 797.152466] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 797.422149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 797.430601] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 797.718165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 797.740906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 797.814598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.822150] LNetError: 114077:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.824545] LNet: Removed LNI 192.168.204.29@tcp [ 797.835242] LNet: Removed LNI 192.168.204.29@tcp2 [ 797.842499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.862543] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 797.864945] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 797.867478] LNet: 114083:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 798.006860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 798.073178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.136129] LNet: Removed LNI 192.168.204.29@tcp [ 799.137559] LNetError: 114120:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.139665] LNetError: 114120:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 799.147322] LNet: Removed LNI 192.168.204.29@tcp2 [ 799.156422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.180851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 799.189732] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 799.455850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 799.465278] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 799.772451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 799.790221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 799.810510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 799.878860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.885903] LNet: 114404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.888062] LNet: 114404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 800.928170] LNet: Removed LNI 192.168.204.29@tcp [ 801.958337] LNet: Removed LNI 192.168.204.29@tcp2 [ 801.966461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.987293] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 801.991218] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 801.993178] LNet: 114411:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 802.144587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 802.213207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.221379] LNetError: 114448:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.224496] LNetError: 114448:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 802.228303] LNet: Removed LNI 192.168.204.29@tcp [ 803.238276] LNet: Removed LNI 192.168.204.29@tcp2 [ 803.247295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.269857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 803.282465] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 803.545112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 803.553236] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 803.896970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 803.924409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 804.031812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 804.041234] LNet: 114727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.043268] LNet: 114727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 805.088093] LNet: Removed LNI 192.168.204.29@tcp [ 805.096627] LNet: Removed LNI 192.168.204.29@tcp2 [ 805.107915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.143948] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 805.147658] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 805.151428] LNet: 114733:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 805.327846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 805.404252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.432147] LNet: Removed LNI 192.168.204.29@tcp [ 806.433401] LNetError: 114770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.435342] LNetError: 114770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 806.443491] LNet: Removed LNI 192.168.204.29@tcp2 [ 806.451113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.469316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 806.480709] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 806.727693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 806.734928] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 807.039384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 807.054257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 807.072252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 807.148828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.160112] LNet: Removed LNI 192.168.204.29@tcp [ 809.190355] LNet: Removed LNI 192.168.204.29@tcp2 [ 809.199382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.221794] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 809.224147] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 809.225727] LNet: 115061:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 809.369623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 809.443625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.452133] LNet: Removed LNI 192.168.204.29@tcp [ 810.470354] LNet: Removed LNI 192.168.204.29@tcp2 [ 810.478273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.493687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 810.500576] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 810.714182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 810.722089] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 810.975522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 810.986428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 811.004349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 811.113169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.128117] LNet: Removed LNI 192.168.204.29@tcp [ 812.135279] LNet: Removed LNI 192.168.204.29@tcp2 [ 812.143229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.160806] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 812.162799] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 812.301308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 812.409544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.417222] LNet: 115452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.419192] LNet: 115452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 813.472128] LNet: Removed LNI 192.168.204.29@tcp [ 813.479268] LNet: Removed LNI 192.168.204.29@tcp2 [ 813.486957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.508508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 813.517181] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 813.753100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 813.759969] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 814.026813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 814.039413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 814.049281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 814.067186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 814.177537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.200128] LNet: Removed LNI 192.168.204.29@tcp [ 815.201671] LNetError: 115753:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.204288] LNetError: 115753:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 815.213262] LNet: Removed LNI 192.168.204.29@tcp2 [ 815.221399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.243179] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 815.246566] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 815.249435] LNet: 115759:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 815.252624] LNet: 115759:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 815.399741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 817.542575] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 818.040179] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 20:42:55 (1782866575) [ 818.073413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 818.081719] LNet: Removed LNI 192.168.204.29@tcp [ 819.110546] LNet: Removed LNI 192.168.204.29@tcp2 [ 819.117727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.204881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 819.213962] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 819.453524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 819.462692] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 819.739753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.749500] LNet: Removed LNI 10.1.2.3@tcp [ 819.757303] LNet: Removed LNI 192.168.204.29@tcp [ 819.764582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.789668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 819.800481] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 819.802297] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 819.933859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 819.942667] LNet: Removed LNI 192.168.204.29@tcp [ 819.955252] LNet: Removed LNI 10.1.2.3@tcp [ 821.860304] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 20:42:58 (1782866578) [ 821.908896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.921328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.951760] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 821.996968] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 821.998568] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 822.000319] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 824.128679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 824.137802] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 824.363445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 824.371782] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 824.997884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 825.011304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 825.031037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.29@tcp -m PUT -e local_timeout -r 1 [ 825.053255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 825.070953] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 825.072743] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 825.139859] LNetError: 117457:0:(lib-move.c:4714:lnet_parse()) 192.168.204.29@tcp, src 192.168.204.29@tcp: Dropping PUT (error -108 looking up sender) [ 826.144508] LNet: Removed LNI 10.1.2.3@tcp [ 827.174265] LNet: Removed LNI 192.168.204.29@tcp [ 827.312361] Key type .llcrypt unregistered [ 827.313385] Key type ._llcrypt unregistered [ 830.773821] Key type ._llcrypt registered [ 830.775088] Key type .llcrypt registered [ 831.196388] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 20:43:08 (1782866588) [ 831.265731] LNet: 118685:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 831.265731] mask usage: [+|-] ... [ 831.313037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 831.317851] alg: No test for adler32 (adler32-zlib) [ 832.121594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.246721] LNet: 118766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.455657] Key type .llcrypt unregistered [ 832.456767] Key type ._llcrypt unregistered [ 834.352723] Key type ._llcrypt registered [ 834.353855] Key type .llcrypt registered [ 834.365307] LNet: 118994:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 834.365307] mask usage: [+|-] ... [ 834.412883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 834.418301] alg: No test for adler32 (adler32-zlib) [ 835.220489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 835.231851] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 838.036251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 838.086762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 838.281915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.129@tcp [ 838.304091] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.129@tcp [ 838.355862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 838.366047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 838.375901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 838.385650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 838.396818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 838.406887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 838.417246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 838.427318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 838.434297] LNet: 119214:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 838.441955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 838.453405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 838.464763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 838.475738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 838.486417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 838.498646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 838.703368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 838.779120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 838.790841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 838.803537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 838.815505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 838.827013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 838.837845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 838.850044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 838.860291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 838.871956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 838.882273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 838.893723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 838.904286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 838.915468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 838.926191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 839.125872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.129@tcp [ 839.203719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 839.214104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 839.224858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 839.235037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 839.244426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 839.254290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 839.264297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 839.273931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 839.280015] LNet: 119448:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 839.282315] LNet: 119448:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 839.288608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 839.299551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 839.311906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 839.322573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 839.333267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 839.344685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 839.550894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 839.626987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 839.637509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 839.648584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 839.659095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 839.670960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 839.682609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 839.693775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 839.706793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 839.718618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 839.731404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 839.743415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 839.755388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 839.767397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 839.781684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 839.894665] LNet: 119611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.897277] LNetError: 119611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.906326] LNet: Removed LNI 192.168.204.29@tcp [ 840.055529] Key type .llcrypt unregistered [ 840.056620] Key type ._llcrypt unregistered [ 845.023367] Key type ._llcrypt registered [ 845.024348] Key type .llcrypt registered [ 845.465240] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 20:43:22 (1782866602) [ 845.545298] LNet: 120415:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 845.545298] mask usage: [+|-] ... [ 845.593897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 845.599049] alg: No test for adler32 (adler32-zlib) [ 846.397502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.515652] LNet: 120496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.722446] Key type .llcrypt unregistered [ 846.723511] Key type ._llcrypt unregistered [ 848.724457] Key type ._llcrypt registered [ 848.726395] Key type .llcrypt registered [ 848.740377] LNet: 120724:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 848.740377] mask usage: [+|-] ... [ 848.793233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 848.799422] alg: No test for adler32 (adler32-zlib) [ 849.600751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 849.610646] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 852.428236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 854.985662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.000084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 855.011517] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 855.353503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 855.645151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.129@tcp [ 855.753398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 855.765573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 855.776126] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.129@tcp [ 855.777343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 855.789694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 855.801152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 855.811856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 855.823202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 855.833655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 855.839657] LNet: 121146:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 855.846060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 855.861176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 855.876572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 855.890658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 855.905951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 855.923753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 856.235341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 856.364652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 856.382538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 856.399781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 856.419836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 856.438809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 856.455659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 856.473416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 856.488431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 856.498374] LNet: 121263:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 856.509072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 856.523682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 856.536631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 856.548597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 856.560926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 856.573181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 856.839978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.129@tcp [ 856.955586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 856.971470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 856.986275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 856.999961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 857.013412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 857.026944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 857.045960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 857.063522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 857.079338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 857.095264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 857.111492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 857.129283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 857.149062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 857.166268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 857.466301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 857.587406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 857.604912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 857.617015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 857.629349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 857.644092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 857.658019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 857.671680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 857.686871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 857.696468] LNet: 121497:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 857.699859] LNet: 121497:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 857.707926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 857.720963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 857.736878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 857.754169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 857.770597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 857.786822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 860.120145] LNet: 121577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.124151] LNetError: 121577:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.129892] LNet: Removed LNI 192.168.204.29@tcp [ 860.138405] LNet: Removed LNI 192.168.204.29@tcp1 [ 860.364425] Key type .llcrypt unregistered [ 860.367090] Key type ._llcrypt unregistered [ 865.504421] Key type ._llcrypt registered [ 865.505570] Key type .llcrypt registered [ 866.014551] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 20:43:42 (1782866622) [ 868.322028] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 20:43:45 (1782866625) [ 870.801312] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 20:43:47 (1782866627) [ 873.420898] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 20:43:50 (1782866630) [ 876.086990] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 20:43:52 (1782866632) [ 878.549785] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 20:43:55 (1782866635) [ 881.085642] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 20:43:58 (1782866638) [ 883.786239] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 20:44:00 (1782866640) [ 886.250779] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 20:44:03 (1782866643) [ 888.715204] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 20:44:05 (1782866645) [ 890.974299] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 20:44:07 (1782866647) [ 893.290836] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 20:44:10 (1782866650) [ 895.803706] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 20:44:12 (1782866652) [ 898.068813] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 20:44:15 (1782866655) [ 900.296649] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 20:44:17 (1782866657) [ 902.512928] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 20:44:19 (1782866659) [ 904.836098] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 20:44:21 (1782866661) [ 907.293868] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 20:44:24 (1782866664) [ 907.370362] LNet: 132102:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 907.370362] mask usage: [+|-] ... [ 907.418564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 907.423266] alg: No test for adler32 (adler32-zlib) [ 908.221415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.264910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 908.277573] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 908.802408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.809103] LNet: 132273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.811653] LNetError: 132273:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.820251] LNet: Removed LNI 192.168.204.29@tcp [ 908.825927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.841806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 908.849492] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 911.284581] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 20:44:28 (1782866668) [ 914.135896] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 20:44:31 (1782866671) [ 916.696941] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 20:44:33 (1782866673) [ 919.485342] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 20:44:36 (1782866676) [ 921.910316] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 20:44:38 (1782866678) [ 924.123550] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 20:44:41 (1782866681) [ 926.458696] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 20:44:43 (1782866683) [ 928.854489] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 20:44:45 (1782866685) [ 931.294076] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 20:44:48 (1782866688) [ 934.012725] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 20:44:50 (1782866690) [ 936.520171] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 20:44:53 (1782866693) [ 938.857451] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 20:44:55 (1782866695) [ 941.150131] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 842 sec =========== 20:44:58 (1782866698) [ 941.296443] LNet: 139728:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.299135] LNetError: 139728:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.306614] LNet: Removed LNI 192.168.204.29@tcp [ 941.507590] Key type .llcrypt unregistered [ 941.508809] Key type ._llcrypt unregistered [ 945.964947] Key type ._llcrypt registered [ 945.966260] Key type .llcrypt registered [ 946.043674] LNet: 140212:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 946.043674] mask usage: [+|-] ... [ 946.131461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 946.135927] alg: No test for adler32 (adler32-zlib) [ 946.992116] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 947.074756] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 948.664175] Key type lgssc registered [ 949.038490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 975.607607] Lustre: Mounted lustre-client [ 975.662551] LNet: 141678:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 975.662551] mask usage: [+|-] ... [ 978.156522] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 980.713054] Lustre: Unmounted lustre-client [ 1006.601440] Key type lgssc unregistered [ 1006.736401] LNet: 142661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.739355] LNetError: 142661:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.747680] LNet: Removed LNI 192.168.204.29@tcp [ 1006.985121] Key type .llcrypt unregistered [ 1006.986306] Key type ._llcrypt unregistered