[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 424840054 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002328] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.006917] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.010112] LSM: Security Framework initializing [ 0.011035] Yama: becoming mindful. [ 0.011753] SELinux: Initializing. [ 0.012046] *** VALIDATE selinux *** [ 0.019664] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023611] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024120] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027016] *** VALIDATE tmpfs *** [ 0.028546] *** VALIDATE proc *** [ 0.029254] *** VALIDATE cgroup *** [ 0.030007] *** VALIDATE cgroup2 *** [ 0.031286] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032134] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034034] Spectre V2 : User space: Vulnerable [ 0.035007] Speculative Store Bypass: Vulnerable [ 0.038579] debug: unmapping init [mem 0xffffffffa6c59000-0xffffffffa6c60fff] [ 0.040176] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041720] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042023] ... version: 2 [ 0.043012] ... bit width: 48 [ 0.044010] ... generic registers: 4 [ 0.045012] ... value mask: 0000ffffffffffff [ 0.046014] ... max period: 00007fffffffffff [ 0.047014] ... fixed-purpose events: 3 [ 0.048011] ... event mask: 000000070000000f [ 0.049323] rcu: Hierarchical SRCU implementation. [ 0.051500] smp: Bringing up secondary CPUs ... [ 0.052863] x86: Booting SMP configuration: [ 0.053027] .... node #0, CPUs: #1 #2 #3 [ 0.056297] smp: Brought up 1 node, 4 CPUs [ 0.058013] smpboot: Max logical packages: 1 [ 0.059013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.117020] node 0 deferred pages initialised in 55ms [ 0.120098] devtmpfs: initialized [ 0.121218] x86/mm: Memory block size: 128MB [ 0.124000] gcov: version magic: 0x41383552 [ 0.126292] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.127069] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.128271] pinctrl core: initialized pinctrl subsystem [ 0.129188] [ 0.129771] ************************************************************* [ 0.130012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.131008] ** ** [ 0.132007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.133025] ** ** [ 0.134008] ** This means that this kernel is built to expose internal ** [ 0.135007] ** IOMMU data structures, which may compromise security on ** [ 0.136009] ** your system. ** [ 0.137010] ** ** [ 0.138006] ** If you see this message and you are not debugging the ** [ 0.139005] ** kernel, report this immediately to your vendor! ** [ 0.140010] ** ** [ 0.141008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.142011] ************************************************************* [ 0.144000] NET: Registered protocol family 16 [ 0.145736] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.148089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.151092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.155016] cpuidle: using governor menu [ 0.156666] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.159487] PCI: Using configuration type 1 for base access [ 0.161119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.170139] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.172053] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.176107] cryptd: max_cpu_qlen set to 1000 [ 0.179419] ACPI: Added _OSI(Module Device) [ 0.181028] ACPI: Added _OSI(Processor Device) [ 0.182019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.184020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.189465] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.197320] ACPI: Interpreter enabled [ 0.198067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.200014] ACPI: Using IOAPIC for interrupt routing [ 0.201109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.204485] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.215058] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.218078] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.221020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.224091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.229497] acpiphp: Slot [2] registered [ 0.230113] acpiphp: Slot [5] registered [ 0.232102] acpiphp: Slot [6] registered [ 0.233085] acpiphp: Slot [3] registered [ 0.235088] acpiphp: Slot [4] registered [ 0.236101] acpiphp: Slot [7] registered [ 0.237098] acpiphp: Slot [8] registered [ 0.239085] acpiphp: Slot [9] registered [ 0.240104] acpiphp: Slot [10] registered [ 0.242102] acpiphp: Slot [11] registered [ 0.243083] acpiphp: Slot [12] registered [ 0.244085] acpiphp: Slot [13] registered [ 0.246086] acpiphp: Slot [14] registered [ 0.247085] acpiphp: Slot [15] registered [ 0.248105] acpiphp: Slot [16] registered [ 0.249148] acpiphp: Slot [17] registered [ 0.251076] acpiphp: Slot [18] registered [ 0.252109] acpiphp: Slot [19] registered [ 0.253089] acpiphp: Slot [20] registered [ 0.255081] acpiphp: Slot [21] registered [ 0.256135] acpiphp: Slot [22] registered [ 0.257107] acpiphp: Slot [23] registered [ 0.259094] acpiphp: Slot [24] registered [ 0.260099] acpiphp: Slot [25] registered [ 0.262100] acpiphp: Slot [26] registered [ 0.263090] acpiphp: Slot [27] registered [ 0.265117] acpiphp: Slot [28] registered [ 0.266096] acpiphp: Slot [29] registered [ 0.268097] acpiphp: Slot [30] registered [ 0.269091] acpiphp: Slot [31] registered [ 0.271096] PCI host bridge to bus 0000:00 [ 0.272017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.274022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.277021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.279022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.282026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.284028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.286219] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.290075] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.292770] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.299016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.303060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.306018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.307012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.309018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.311570] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.315014] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.317038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.318805] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.322824] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.330013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.332015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.337567] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.351019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.358014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.375020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.383662] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.388012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.393037] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.409020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.419116] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.423364] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.425343] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.427350] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.429369] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.434158] iommu: Default domain type: Passthrough [ 0.435807] SCSI subsystem initialized [ 0.437116] ACPI: bus type USB registered [ 0.438111] usbcore: registered new interface driver usbfs [ 0.440116] usbcore: registered new interface driver hub [ 0.442080] usbcore: registered new device driver usb [ 0.445178] pps_core: LinuxPPS API ver. 1 registered [ 0.446011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.449082] PTP clock support registered [ 0.451193] EDAC MC: Ver: 3.0.0 [ 0.453137] PCI: Using ACPI for IRQ routing [ 0.454537] NetLabel: Initializing [ 0.456010] NetLabel: domain hash size = 128 [ 0.457016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.459074] NetLabel: unlabeled traffic allowed by default [ 0.460112] vgaarb: loaded [ 0.461193] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.462007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.469017] clocksource: Switched to clocksource kvm-clock [ 0.587001] VFS: Disk quotas dquot_6.6.0 [ 0.588404] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.590984] *** VALIDATE ramfs *** [ 0.592416] *** VALIDATE hugetlbfs *** [ 0.594212] pnp: PnP ACPI init [ 0.596692] pnp: PnP ACPI: found 6 devices [ 0.612817] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.615768] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.617838] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.620022] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.622302] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.624636] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.626994] NET: Registered protocol family 2 [ 0.629160] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.633865] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.637409] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.641896] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.644801] TCP: Hash tables configured (established 65536 bind 65536) [ 0.647250] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.649828] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.652327] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.654851] NET: Registered protocol family 1 [ 0.657243] RPC: Registered named UNIX socket transport module. [ 0.660041] RPC: Registered udp transport module. [ 0.662036] RPC: Registered tcp transport module. [ 0.664061] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.666727] NET: Registered protocol family 44 [ 0.668620] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.671109] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.673401] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.676240] PCI: CLS 0 bytes, default 64 [ 0.678153] Unpacking initramfs... [ 2.046825] debug: unmapping init [mem 0xffff961a7cc64000-0xffff961a7ffcffff] [ 2.050065] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.051734] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.053854] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.513955] Initialise system trusted keyrings [ 2.515309] Key type blacklist registered [ 2.517109] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.527083] zbud: loaded [ 2.530293] *** VALIDATE nfs *** [ 2.531245] *** VALIDATE nfs4 *** [ 2.532737] pstore: using deflate compression [ 2.536694] Platform Keyring initialized [ 2.625551] NET: Registered protocol family 38 [ 2.627582] Key type asymmetric registered [ 2.629500] Asymmetric key parser 'x509' registered [ 2.631515] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.634566] io scheduler mq-deadline registered [ 2.635882] io scheduler kyber registered [ 2.637092] io scheduler bfq registered [ 2.638396] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.640565] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.643111] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.646310] ACPI: Power Button [PWRF] [ 2.652076] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.658558] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.667990] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.695743] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.724808] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.729475] Non-volatile memory driver v1.3 [ 2.730808] Linux agpgart interface v0.103 [ 2.759583] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 2.763130] vda: detected capacity change from 0 to 74645504 [ 2.782195] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.785219] vdb: detected capacity change from 0 to 1073741824 [ 2.791975] libphy: Fixed MDIO Bus: probed [ 2.799196] usbcore: registered new interface driver usbserial_generic [ 2.801553] usbserial: USB Serial support registered for generic [ 2.803783] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.808419] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.810461] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.814294] mousedev: PS/2 mouse device common for all mice [ 2.817531] rtc_cmos 00:05: RTC can wake from S4 [ 2.820088] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.823367] rtc_cmos 00:05: registered as rtc0 [ 2.825144] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.827978] intel_pstate: CPU model not supported [ 2.830487] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.834252] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.834309] hid: raw HID events driver (C) Jiri Kosina [ 2.838410] usbcore: registered new interface driver usbhid [ 2.840100] usbhid: USB HID core driver [ 2.841449] drop_monitor: Initializing network drop monitor service [ 2.843508] Initializing XFRM netlink socket [ 2.845329] NET: Registered protocol family 10 [ 2.848415] Segment Routing with IPv6 [ 2.849916] NET: Registered protocol family 17 [ 2.852722] mpls_gso: MPLS GSO support [ 2.858449] RAS: Correctable Errors collector initialized. [ 2.860446] AVX version of gcm_enc/dec engaged. [ 2.861751] AES CTR mode by8 optimization enabled [ 2.940679] sched_clock: Marking stable (2940645846, 0)->(3796607484, -855961638) [ 2.946676] registered taskstats version 1 [ 2.948397] Loading compiled-in X.509 certificates [ 2.951607] zswap: loaded using pool lzo/zbud [ 2.977262] Key type big_key registered [ 2.991651] Key type encrypted registered [ 2.993015] ima: No TPM chip found, activating TPM-bypass! [ 2.994383] ima: Allocated hash algorithm: sha1 [ 2.996217] ima: No architecture policies found [ 2.997947] evm: Initialising EVM extended attributes: [ 2.999728] evm: security.selinux [ 3.000738] evm: security.ima [ 3.001773] evm: security.capability [ 3.002938] evm: HMAC attrs: 0x1 [ 3.004942] rtc_cmos 00:05: setting system clock to 2026-07-16 18:01:41 UTC (1784224901) [ 3.010360] debug: unmapping init [mem 0xffffffffa7c03000-0xffffffffa7dfffff] [ 3.012602] debug: unmapping init [mem 0xffffffffa6982000-0xffffffffa6c58fff] [ 3.021182] Write protecting the kernel read-only data: 28672k [ 3.024525] debug: unmapping init [mem 0xffffffffa5003000-0xffffffffa51fffff] [ 3.026921] debug: unmapping init [mem 0xffffffffa5914000-0xffffffffa59fffff] [ 3.057273] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.065100] systemd[1]: Detected virtualization kvm. [ 3.066801] systemd[1]: Detected architecture x86-64. [ 3.068501] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.091782] systemd[1]: No hostname configured. [ 3.093034] systemd[1]: Set hostname to . [ 3.094335] random: systemd: uninitialized urandom read (16 bytes read) [ 3.096643] systemd[1]: Initializing machine ID from random generator. [ 3.145365] random: ln: uninitialized urandom read (6 bytes read) [ 3.238531] random: systemd: uninitialized urandom read (16 bytes read) [ 3.241143] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.245056] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.247986] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.824033] device-mapper: uevent: version 1.0.3 [ 3.825441] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook.[ 4.189095] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.564828] virtio_net virtio0 ens2: renamed from eth0 [ 4.626734] scsi host0: ata_piix [ 4.647958] scsi host1: ata_piix [ 4.649799] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.652275] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.035222] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.511103] random: crng init done [ 9.512584] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.912896] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.036614] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.290455] SELinux: Disabled at runtime. [ 11.353039] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.366111] systemd[1]: Detected virtualization kvm. [ 11.368186] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.813326] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.815509] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.819354] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.822961] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.825470] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.836311] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.845367] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ 11.902494] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.271747] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ OK ] Started udev Coldplug all Devices. [ 12.631091] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.659592] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.781306] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.795981] EDAC sbridge: Ver: 1.1.2 [ 13.939836] Key type dns_resolver registered [ 14.236809] NFS: Registering the id_resolver key type [ 14.239499] Key type id_resolver registered [ 14.241463] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg146-client login: [ 48.090781] libcfs: loading out-of-tree module taints kernel. [ 48.205739] Key type ._llcrypt registered [ 48.208369] Key type .llcrypt registered [ 48.465178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 48.475327] alg: No test for adler32 (adler32-zlib) [ 49.657255] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 50.133618] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 51.815354] Key type lgssc registered [ 52.669933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 124.979880] Lustre: Mounted lustre-client [ 128.652418] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 137.818561] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing check_logdir /tmp/testlogs/ [ 140.696082] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing yml_node [ 143.505223] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 145.063814] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 146.937922] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 147.828655] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Jul 16 14:04:05 EDT 2026 [ 150.495239] Lustre: lustre-OST0000-osc-ffff961ac3efc000: disconnect after 23s idle [ 157.740737] Lustre: Unmounted lustre-client [ 169.445142] Key type lgssc unregistered [ 169.607986] LNet: 8371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 169.613347] LNetError: 8371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 169.629632] LNet: Removed LNI 192.168.201.46@tcp [ 170.053295] Key type .llcrypt unregistered [ 170.056483] Key type ._llcrypt unregistered [ 183.865794] Key type ._llcrypt registered [ 183.867154] Key type .llcrypt registered [ 183.986322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 183.997320] alg: No test for adler32 (adler32-zlib) [ 184.834678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 184.873582] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 184.885185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 187.177354] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 187.263580] LNet: 9244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 187.268347] LNetError: 9244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 187.286758] LNet: Removed LNI 192.168.201.46@tcp [ 187.757452] Key type .llcrypt unregistered [ 187.759357] Key type ._llcrypt unregistered [ 191.554971] Key type ._llcrypt registered [ 191.556487] Key type .llcrypt registered [ 192.327457] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 14:04:49 (1784225089) [ 192.499687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 192.508328] alg: No test for adler32 (adler32-zlib) [ 193.322888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 193.365703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 193.380283] LNet: 9662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 193.391592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 193.419230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 193.431910] LNet: 9669:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 197.149837] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 14:04:54 (1784225094) [ 197.284502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 197.295964] LNet: 10246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 197.309133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 197.355808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 201.453411] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 14:04:59 (1784225099) [ 201.503147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 201.523285] LNet: 10827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 201.537806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 201.583624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 205.347657] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 14:05:03 (1784225103) [ 205.405588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 205.423655] LNet: 11407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 205.440425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 205.495982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 209.243935] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 14:05:06 (1784225106) [ 209.288397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 209.310686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 209.357659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 209.579240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 213.511719] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 14:05:11 (1784225111) [ 213.594648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 213.611607] LNet: 12586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 213.615681] LNet: 12586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 213.632775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 213.680365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 213.908649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 214.176458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 217.499965] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 14:05:15 (1784225115) [ 217.678740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 217.701950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 217.752474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 222.011476] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 14:05:19 (1784225119) [ 222.216818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.233541] LNet: 13767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 222.237630] LNet: 13767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 222.252793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.298571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 222.325891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 222.582252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 222.603956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 222.803630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 222.825345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 223.086485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 223.102484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 223.306498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 223.333319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 223.546918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 223.576967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 223.821437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 223.841785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 224.043460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 224.065464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 224.282756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 224.303483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 224.534968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 224.555677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 224.797973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 224.819077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 225.050856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 225.082200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 225.346361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 225.385367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 228.781480] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 14:05:26 (1784225126) [ 228.889828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 228.915663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 228.958240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 228.979797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 232.105859] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 14:05:29 (1784225129) [ 232.187559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 232.205203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 232.239810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 232.260680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 235.551500] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 14:05:33 (1784225133) [ 235.604370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.626216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.677484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 235.701594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 238.763334] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 14:05:36 (1784225136) [ 238.869444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.878699] LNet: 16205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 238.881933] LNet: 16205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 238.893264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.944237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 238.962989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 241.799236] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 14:05:39 (1784225139) [ 241.870284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 241.890196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 241.926718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 241.949223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 245.392236] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 14:05:43 (1784225143) [ 245.448098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 245.492564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 245.564496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 245.584218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 248.944579] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 14:05:46 (1784225146) [ 249.077504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.093756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.139180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 249.349432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 253.360706] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 14:05:51 (1784225151) [ 253.449528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 253.496372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 253.638253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 253.776795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 257.872669] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 14:05:55 (1784225155) [ 257.988638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.007951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.082753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 258.241960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 262.060115] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 14:05:59 (1784225159) [ 262.114812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 262.145983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 262.526838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 262.684730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 266.422594] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 14:06:04 (1784225164) [ 266.471837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 266.492363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 266.541662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 266.573292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 269.563210] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 14:06:07 (1784225167) [ 269.676935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 269.694760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 269.731460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 269.761274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 272.933407] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 14:06:10 (1784225170) [ 273.053901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.065631] LNet: 21641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 273.068259] LNet: 21641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 273.078292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.129110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 276.231702] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 14:06:14 (1784225174) [ 276.424667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 276.445954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.503881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 280.238303] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 14:06:18 (1784225178) [ 280.379606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 280.395187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 280.431605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 280.606883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 283.938331] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 14:06:21 (1784225181) [ 284.026710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 284.041710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 284.072537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 284.093295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 287.051273] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 14:06:24 (1784225184) [ 287.170581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 287.197462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 287.236872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 287.253796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 290.290128] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 14:06:28 (1784225188) [ 290.425363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.454254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.510964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 290.541932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 293.992734] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 14:06:31 (1784225191) [ 294.139319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 294.153331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.185790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 294.201779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 297.181160] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 14:06:35 (1784225195) [ 297.256974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.288675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.337966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 297.364305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 297.374583] LNetError: 25753:0:(peer.c:2276:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 297.394867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 297.409621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 300.414848] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 14:06:38 (1784225198) [ 300.611912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.632102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.679988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.HZdQ3b4Ooa [ 300.726388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.HZdQ3b4Ooa [ 300.761700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.HZdQ3b4Ooa [ 303.663987] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 14:06:41 (1784225201) [ 303.781793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.799353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.826628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 303.848435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 303.875464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 307.248450] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 14:06:44 (1784225204) [ 307.358085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.390736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.436568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 307.457086] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 308.286844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 309.012286] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 310.767346] Key type lgssc registered [ 311.338960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 313.140588] LNetError: 27512:0:(lib-move.c:2120:lnet_handle_find_routed_path()) peer 192.168.201.46@tcp99 has no available nets [ 314.479515] Key type lgssc unregistered [ 314.699119] LNetError: 28619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 314.706835] LNet: Removed LNI 192.168.201.46@tcp [ 315.012614] Key type .llcrypt unregistered [ 315.014699] Key type ._llcrypt unregistered [ 318.210702] Key type ._llcrypt registered [ 318.211864] Key type .llcrypt registered [ 318.990466] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 14:06:56 (1784225216) [ 319.289254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 319.296990] alg: No test for adler32 (adler32-zlib) [ 320.136348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.187883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 321.473229] LNet: Message forwarding will be enabled in 2s [ 323.554638] LNet: Message forwarding enabled [ 323.645995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 323.657357] LNet: Message forwarding will stop in 2s to 6s [ 327.941965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 328.293108] LNet: Message forwarding disabled [ 329.026799] LNet: Message forwarding will be enabled in 2s [ 332.383075] LNet: Message forwarding enabled [ 333.062562] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 14:07:10 (1784225230) [ 333.232676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.250343] LNet: 30016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.884741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.927965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 337.371213] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 14:07:15 (1784225235) [ 337.548350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.561953] LNet: 30597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.581695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.612888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 337.637527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 337.657796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 337.674666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 337.694564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 337.712319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 337.727536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 337.748550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 337.764694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 337.783075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 337.799878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 337.815921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 337.832455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 337.849747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 337.868752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 337.884796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 337.904373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 337.924770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 337.945170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 337.957598] LNetError: 30640:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 341.041111] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 14:07:18 (1784225238) [ 341.203103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 341.213637] LNet: 31212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.223969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.265393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 344.670319] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 14:07:22 (1784225242) [ 345.674984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg146-server [ 346.587064] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 14:07:24 (1784225244) [ 346.680040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.693341] LNet: 32143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.703057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.729768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 346.742749] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 347.213405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[47-51]@tcp [ 347.232501] LNet: 32276:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 347.299916] LNetError: 32147:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.47@tcp is being used as a gateway but routing feature is not turned on [ 347.312842] LNetError: 32147:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.47@tcp has gone from up to down [ 348.257505] LNetError: 32147:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.49@tcp is being used as a gateway but routing feature is not turned on [ 348.269545] LNetError: 32147:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 349.283800] LNetError: 32147:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.47@tcp is being used as a gateway but routing feature is not turned on [ 349.295474] LNetError: 32147:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 350.312387] LNetError: 32152:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.48@tcp has gone from up to down [ 350.322285] LNetError: 32152:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 351.113728] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 14:07:28 (1784225248) [ 351.397826] LNetError: 32147:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.47@tcp is being used as a gateway but routing feature is not turned on [ 351.410176] LNetError: 32147:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 352.194504] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg146-server [ 353.363444] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 14:07:30 (1784225250) [ 353.438372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.456577] LNet: 33204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.459580] LNetError: 33204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 353.468414] LNet: Removed LNI 192.168.201.46@tcp [ 353.483478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.532431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 353.549825] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 353.936869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[47-52/2]@tcp [ 353.954672] LNet: 33328:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 353.963187] LNet: 33328:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 353.980397] LNetError: 33215:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.47@tcp has gone from up to down [ 353.980774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[47-52/2]@tcp [ 357.929261] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 14:07:35 (1784225255) [ 358.103656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.119088] LNetError: 33921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 358.143611] LNet: Removed LNI 192.168.201.46@tcp [ 358.163848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.217563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 358.277124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.314853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.359215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 358.399938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.439621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.476352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 358.531909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.556591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.599238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 358.644644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.664464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.711530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 358.777238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.802570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.839366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 358.897809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.933147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.955275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 359.004721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.022333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.061477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 359.124960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.153692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.195557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 359.251459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.281471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.331842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 359.381527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.404716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.432166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 359.462399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.486079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.519312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 363.364356] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 14:07:41 (1784225261) [ 364.565424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg146-server [ 365.587745] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 14:07:43 (1784225263) [ 366.403818] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg146-server [ 367.364472] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 14:07:45 (1784225265) [ 367.481303] LNet: 35320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.485620] LNet: 35320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 367.798976] Key type .llcrypt unregistered [ 367.799932] Key type ._llcrypt unregistered [ 370.817165] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 370.909450] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 370.916772] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 370.923428] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 373.195415] Key type ._llcrypt registered [ 373.196612] Key type .llcrypt registered [ 373.299204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 373.309207] alg: No test for adler32 (adler32-zlib) [ 374.181752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.228176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 374.244260] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 374.682194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 374.692588] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 375.785292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 375.800603] LNet: Removed LNI 10.1.2.3@tcp [ 375.886889] LNet: 36041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.892888] LNetError: 36041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 375.901800] LNet: Removed LNI 192.168.201.46@tcp [ 376.322428] Key type .llcrypt unregistered [ 376.324186] Key type ._llcrypt unregistered [ 382.698177] Key type ._llcrypt registered [ 382.699990] Key type .llcrypt registered [ 383.427905] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 14:08:01 (1784225281) [ 383.759554] Key type .llcrypt unregistered [ 383.762589] Key type ._llcrypt unregistered [ 386.911874] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 387.001211] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 387.004475] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 387.008169] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 389.256840] Key type ._llcrypt registered [ 389.258645] Key type .llcrypt registered [ 389.340819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 389.349497] alg: No test for adler32 (adler32-zlib) [ 390.183277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.217725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 390.239908] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 390.594372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 390.606119] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 391.790915] LNet: 37531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.793868] LNetError: 37531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 391.796044] LNet: Removed LNI 10.1.2.3@tcp [ 391.805621] LNet: Removed LNI 192.168.201.46@tcp [ 392.072383] Key type .llcrypt unregistered [ 392.074302] Key type ._llcrypt unregistered [ 397.344395] Key type ._llcrypt registered [ 397.346012] Key type .llcrypt registered [ 398.072614] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 14:08:15 (1784225295) [ 398.363234] Key type .llcrypt unregistered [ 398.364271] Key type ._llcrypt unregistered [ 401.382768] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 401.516897] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 401.527425] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 401.533972] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 403.898223] Key type ._llcrypt registered [ 403.900022] Key type .llcrypt registered [ 404.059272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 404.071968] alg: No test for adler32 (adler32-zlib) [ 404.924954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.957396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 404.985186] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 406.147653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 406.170853] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 407.349524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 407.364971] LNet: Removed LNI 10.1.2.3@tcp [ 407.374589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 407.398990] LNet: Removed LNI 10.1.2.31@tcp [ 407.406589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 407.416167] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 407.807725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 407.822235] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 408.296808] LNet: 39444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.300543] LNetError: 39444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 408.303846] LNet: Removed LNI 10.1.2.31@tcp [ 408.315191] LNet: Removed LNI 10.1.2.3@tcp [ 408.673601] Key type .llcrypt unregistered [ 408.675812] Key type ._llcrypt unregistered [ 413.858307] Key type ._llcrypt registered [ 413.859376] Key type .llcrypt registered [ 414.438933] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 14:08:32 (1784225312) [ 414.630416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 414.636978] alg: No test for adler32 (adler32-zlib) [ 415.452670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.494380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 415.509334] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 415.873249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.884135] LNet: 40396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 415.888344] LNetError: 40396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 415.901360] LNet: Removed LNI 192.168.201.46@tcp [ 415.911505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.939584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.46@tcp [ 415.949174] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 416.320929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.46@tcp [ 416.341140] LNet: Removed LNI 192.168.201.46@tcp [ 416.622196] Key type .llcrypt unregistered [ 416.623396] Key type ._llcrypt unregistered [ 421.717884] Key type ._llcrypt registered [ 421.719401] Key type .llcrypt registered [ 422.276068] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 14:08:40 (1784225320) [ 423.101415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg146-server [ 423.779793] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 14:08:41 (1784225321) [ 424.018390] Key type .llcrypt unregistered [ 424.020246] Key type ._llcrypt unregistered [ 426.619089] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 426.672074] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 426.674018] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 426.676362] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 428.923346] Key type ._llcrypt registered [ 428.924820] Key type .llcrypt registered [ 429.013768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 429.019575] alg: No test for adler32 (adler32-zlib) [ 429.867892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.917259] LNet: Added LNI 192.168.201.46@tcp1 [8/256/0/180] [ 430.321557] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 431.479320] LNet: 42363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 431.481685] LNetError: 42363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 431.484289] LNet: Removed LNI 192.168.201.46@tcp1 [ 431.497895] LNet: Removed LNI 10.1.2.3@tcp2 [ 431.888480] Key type .llcrypt unregistered [ 431.889928] Key type ._llcrypt unregistered [ 438.272908] Key type ._llcrypt registered [ 438.275845] Key type .llcrypt registered [ 439.117059] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 14:08:56 (1784225336) [ 439.370091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 439.381197] alg: No test for adler32 (adler32-zlib) [ 440.223166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.253780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 440.274822] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 440.717932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.728329] LNet: 43331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 440.731202] LNetError: 43331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 440.737358] LNet: Removed LNI 192.168.201.46@tcp [ 440.749405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.783369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 440.796361] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 444.101795] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 14:09:01 (1784225341) [ 444.160537] LNet: 43916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 444.164259] LNetError: 43916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 444.175865] LNet: Removed LNI 192.168.201.46@tcp [ 444.525437] Key type .llcrypt unregistered [ 444.527536] Key type ._llcrypt unregistered [ 447.280354] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 447.349744] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 447.353052] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 447.356143] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 449.627325] Key type ._llcrypt registered [ 449.628556] Key type .llcrypt registered [ 449.704176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 449.712719] alg: No test for adler32 (adler32-zlib) [ 450.556240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.587425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 450.598314] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 450.976400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 450.992925] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 452.194628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.207158] LNet: 44660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 452.212598] LNetError: 44660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 452.218923] LNet: Removed LNI 10.1.2.3@tcp [ 452.224926] LNet: Removed LNI 192.168.201.46@tcp [ 452.242262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.291987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 452.309435] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 452.312075] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 452.541411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.552677] LNet: Removed LNI 10.1.2.3@tcp [ 452.561508] LNet: Removed LNI 192.168.201.46@tcp [ 452.573501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.607299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 452.622232] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 452.625463] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 452.870709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 452.887118] LNet: Added LNI 192.168.201.46@tcp2 [8/256/0/180] [ 453.412180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.424900] LNet: 44849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.428967] LNet: 44849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 453.433507] LNetError: 44849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 453.439745] LNetError: 44849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 453.443651] LNet: Removed LNI 10.1.2.3@tcp [ 453.445016] LNet: Removed LNI 192.168.201.46@tcp [ 453.459754] LNet: Removed LNI 192.168.201.46@tcp2 [ 453.471245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.508737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 453.521535] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 453.524420] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 453.527597] LNet: Added LNI 192.168.201.46@tcp2 [8/256/0/180] [ 456.726110] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 14:09:14 (1784225354) [ 456.879704] LNet: 45446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 456.883113] LNetError: 45446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 456.885679] LNetError: 45446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 456.888426] LNet: Removed LNI 10.1.2.3@tcp [ 456.889478] LNet: Removed LNI 192.168.201.46@tcp [ 456.900341] LNet: Removed LNI 192.168.201.46@tcp2 [ 457.268805] Key type .llcrypt unregistered [ 457.270389] Key type ._llcrypt unregistered [ 460.491570] Key type ._llcrypt registered [ 460.493408] Key type .llcrypt registered [ 460.606246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 460.613659] alg: No test for adler32 (adler32-zlib) [ 461.446639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.482484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 461.498424] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 461.915145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.928331] LNet: 45839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 461.931396] LNetError: 45839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 461.939718] LNet: Removed LNI 192.168.201.46@tcp [ 461.949725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.985643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 462.004204] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 462.222491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.238615] LNet: Removed LNI 192.168.201.46@tcp [ 462.250910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.284135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 462.305027] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 462.501711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.512962] LNet: 45899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 462.515546] LNet: 45899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 462.518546] LNetError: 45899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 462.520939] LNetError: 45899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 462.529048] LNet: Removed LNI 192.168.201.46@tcp [ 462.540249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.574151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 462.585968] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 462.771868] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 462.812252] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 462.814663] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 462.817254] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 465.012974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.022344] LNet: 45978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 465.025271] LNetError: 45978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 465.035423] LNet: Removed LNI 192.168.201.46@tcp [ 465.046991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.082855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 465.097782] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 467.899542] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 14:09:25 (1784225365) [ 467.977977] LNet: 46575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 467.982939] LNetError: 46575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 467.991659] LNet: Removed LNI 192.168.201.46@tcp [ 468.280365] Key type .llcrypt unregistered [ 468.282572] Key type ._llcrypt unregistered [ 470.716802] Key type ._llcrypt registered [ 470.718501] Key type .llcrypt registered [ 470.812706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 470.819487] alg: No test for adler32 (adler32-zlib) [ 471.656951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.692496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 471.703407] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 472.115754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.128398] LNet: 46972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 472.133958] LNetError: 46972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 472.144336] LNet: Removed LNI 192.168.201.46@tcp [ 472.156594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.188547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 472.200704] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 475.101528] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 14:09:33 (1784225373) [ 475.217794] LNet: 47564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 475.220717] LNetError: 47564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 475.231772] LNet: Removed LNI 192.168.201.46@tcp [ 475.534401] Key type .llcrypt unregistered [ 475.536203] Key type ._llcrypt unregistered [ 477.842315] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 477.882592] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 477.885425] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 477.887647] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 480.161517] Key type ._llcrypt registered [ 480.162896] Key type .llcrypt registered [ 480.273485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 480.285348] alg: No test for adler32 (adler32-zlib) [ 481.103235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.129461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 481.141522] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 481.533237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.544251] LNet: 48012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 481.551318] LNetError: 48012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 481.564330] LNet: Removed LNI 192.168.201.46@tcp [ 481.580884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.624934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 481.641928] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 484.325437] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 14:09:42 (1784225382) [ 484.421850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.431527] LNet: 48609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 484.434573] LNetError: 48609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 484.442578] LNet: Removed LNI 192.168.201.46@tcp [ 484.451585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.498917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 486.974482] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 14:09:44 (1784225384) [ 487.012338] LNet: 49182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 487.304665] Key type .llcrypt unregistered [ 487.306730] Key type ._llcrypt unregistered [ 489.770924] Key type ._llcrypt registered [ 489.772342] Key type .llcrypt registered [ 489.845973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 489.851394] alg: No test for adler32 (adler32-zlib) [ 490.673950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.698665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 490.709522] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 491.059582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.069072] LNet: 49587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 491.072340] LNetError: 49587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 491.081488] LNet: Removed LNI 192.168.201.46@tcp [ 491.089388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.116816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 491.129620] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 494.029398] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 14:09:51 (1784225391) [ 494.140090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.151429] LNet: 50180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 494.154432] LNetError: 50180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 494.166967] LNet: Removed LNI 192.168.201.46@tcp [ 494.178532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.222140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 497.185226] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 14:09:55 (1784225395) [ 497.861863] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 498.670476] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 14:09:56 (1784225396) [ 499.415915] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 500.255130] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 14:09:58 (1784225398) [ 500.943374] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 501.806289] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 14:09:59 (1784225399) [ 502.498385] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 503.390365] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 14:10:01 (1784225401) [ 504.169184] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 505.030895] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 14:10:02 (1784225402) [ 505.174778] LNet: 52229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 505.550529] Key type .llcrypt unregistered [ 505.552422] Key type ._llcrypt unregistered [ 508.561951] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 508.613441] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 508.616719] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 508.620602] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 510.985711] Key type ._llcrypt registered [ 510.989136] Key type .llcrypt registered [ 511.130072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 511.138570] alg: No test for adler32 (adler32-zlib) [ 511.957068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.989804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 512.001032] LNet: Added LNI 192.168.201.46@tcp [8/512/0/180] [ 512.004355] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 512.006960] LNet: 52549:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 512.049244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.058390] LNet: 52568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.061449] LNetError: 52568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 512.064157] LNet: Removed LNI 10.1.2.3@tcp [ 512.065623] LNetError: 52568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 512.075678] LNet: Removed LNI 192.168.201.46@tcp [ 512.086167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.119892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 512.131657] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 512.164486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.176301] LNet: 52592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.180048] LNetError: 52592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 512.196839] LNet: Removed LNI 192.168.201.46@tcp [ 512.208084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.239919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 512.252542] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 512.255032] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 512.256642] LNet: 52600:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 512.298267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.310355] LNet: 52618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.315201] LNetError: 52618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 512.318050] LNet: Removed LNI 10.1.2.3@tcp [ 512.319161] LNetError: 52618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 512.325485] LNet: Removed LNI 192.168.201.46@tcp [ 512.335162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.385447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 512.397407] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 512.400832] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 512.403206] LNet: 52626:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 512.484765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.496389] LNet: 52652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.502924] LNetError: 52652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 512.508277] LNet: Removed LNI 10.1.2.3@tcp [ 512.509756] LNetError: 52652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 512.525347] LNet: Removed LNI 192.168.201.46@tcp [ 512.535918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.569750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 512.582356] LNet: Added LNI 192.168.201.46@tcp [8/256/0/182] [ 512.588465] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 512.653476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.663529] LNet: 52680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.667203] LNetError: 52680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 512.670778] LNet: Removed LNI 10.1.2.3@tcp [ 512.672146] LNetError: 52680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 512.681474] LNet: Removed LNI 192.168.201.46@tcp [ 512.693363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.725580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 512.738989] LNet: Added LNI 192.168.201.46@tcp [8/256/0/182] [ 512.742165] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 512.743987] LNet: 52688:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 512.784244] LNet: Removed LNI 10.1.2.3@tcp [ 512.793271] LNet: Removed LNI 192.168.201.46@tcp [ 513.085683] Key type .llcrypt unregistered [ 513.087538] Key type ._llcrypt unregistered [ 518.521178] Key type ._llcrypt registered [ 518.522431] Key type .llcrypt registered [ 519.167180] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 14:10:16 (1784225416) [ 519.350608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 519.356661] alg: No test for adler32 (adler32-zlib) [ 520.182903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.214381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 520.232972] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 520.584319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.46 [ 520.598329] LNet: 53663:0:(config.c:1739:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 520.609048] LNet: Removed LNI 192.168.201.46@tcp [ 523.438895] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 14:10:21 (1784225421) [ 523.546992] LNet: 54233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 523.551470] LNetError: 54233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 523.562794] LNet: Removed LNI 192.168.201.46@tcp [ 523.890418] Key type .llcrypt unregistered [ 523.892867] Key type ._llcrypt unregistered [ 526.494800] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 526.609855] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 528.962533] Key type ._llcrypt registered [ 528.963971] Key type .llcrypt registered [ 529.032840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 529.039389] alg: No test for adler32 (adler32-zlib) [ 530.057970] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 532.704278] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 14:10:30 (1784225430) [ 532.782220] LNet: 55146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 532.788935] LNetError: 55146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 532.798769] LNet: Removed LNI 10.1.2.3@tcp [ 533.035693] Key type .llcrypt unregistered [ 533.037317] Key type ._llcrypt unregistered [ 535.507986] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 535.607329] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 535.609624] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 535.612771] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 537.922655] Key type ._llcrypt registered [ 537.923641] Key type .llcrypt registered [ 537.997657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 538.003798] alg: No test for adler32 (adler32-zlib) [ 538.846444] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 541.662321] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 14:10:39 (1784225439) [ 542.331079] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 543.013918] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 14:10:40 (1784225440) [ 543.072370] LNet: 56324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 543.075561] LNetError: 56324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 543.086231] LNet: Removed LNI 10.1.2.3@tcp [ 543.351604] Key type .llcrypt unregistered [ 543.352806] Key type ._llcrypt unregistered [ 545.843694] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 545.996355] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 545.998653] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 546.001151] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 548.335815] Key type ._llcrypt registered [ 548.337018] Key type .llcrypt registered [ 548.410515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 548.416651] alg: No test for adler32 (adler32-zlib) [ 549.242920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.275590] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 553.295828] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 14:10:51 (1784225451) [ 553.584563] LNet: 57556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 553.587763] LNetError: 57556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 553.604691] LNet: Removed LNI 10.1.2.3@tcp [ 553.880925] Key type .llcrypt unregistered [ 553.882863] Key type ._llcrypt unregistered [ 556.372925] Key type ._llcrypt registered [ 556.374501] Key type .llcrypt registered [ 556.464344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 556.470588] alg: No test for adler32 (adler32-zlib) [ 557.292143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 557.306916] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 560.946742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.146@tcp [ 561.221589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_interrupt -r 1 [ 561.243651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_interrupt -r 1 [ 561.265406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 561.275469] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 561.363503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 561.379475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 561.393283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 561.406977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 561.420096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 561.435032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 561.439089] LNet: 1 local NIs in recovery (showing 1): 192.168.201.46@tcp [ 561.451608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 561.466798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 561.477854] LNet: 57993:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 561.489864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 561.518879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 561.539051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 561.556133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 561.571057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 561.587056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 561.601399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 561.807436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_dropped -r 1 [ 561.823928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_dropped -r 1 [ 561.839942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 561.848982] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 561.930676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 561.943828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 561.959560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 561.974615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 561.987132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 561.999376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 562.013390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 562.029280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 562.040268] LNet: 58102:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 562.049732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 562.071652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 562.093481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 562.110129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 562.126912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 562.142106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 562.157134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 562.374177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_aborted -r 1 [ 562.389747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_aborted -r 1 [ 562.409734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 562.528305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 562.549741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 562.568943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 562.587329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 562.606276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 562.622115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 562.639923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 562.663379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 562.681625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 562.695270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 562.711917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 562.730224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 562.760622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 562.786231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 562.807562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 563.050791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_no_route -r 1 [ 563.074665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_no_route -r 1 [ 563.093673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 563.103044] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 563.105556] LNet: Skipped 2 previous similar messages [ 563.194812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 563.212130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 563.228833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 563.249415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 563.263685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 563.280787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 563.297748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 563.312327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 563.325239] LNet: 58320:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 563.328916] LNet: 58320:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 563.338539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 563.352757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 563.368098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 563.384777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 563.404843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 563.420664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 563.443394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 563.645778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_timeout -r 1 [ 563.664385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_timeout -r 1 [ 563.682655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 563.785889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 563.809168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 563.825978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 563.846043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 563.861693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 563.885408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 563.903709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 563.921281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 563.938675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 563.956110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 563.970704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 563.983772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 563.997488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 564.013368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 564.029723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 564.251946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_error -r 1 [ 564.273993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_error -r 1 [ 564.301325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 564.404698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 564.418886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 564.434138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 564.450293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 564.466108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 564.483793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 564.500226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 564.519388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 564.534408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 564.548561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 564.563707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 564.577397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 564.593224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 564.609828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 564.625376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 564.736256] LNet: 58580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 564.738213] LNetError: 58580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 564.748370] LNet: Removed LNI 192.168.201.46@tcp [ 564.921452] Key type .llcrypt unregistered [ 564.923370] Key type ._llcrypt unregistered [ 571.892551] Key type ._llcrypt registered [ 571.894114] Key type .llcrypt registered [ 572.564053] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 14:11:10 (1784225470) [ 572.932856] Key type .llcrypt unregistered [ 572.936173] Key type ._llcrypt unregistered [ 575.445501] Key type ._llcrypt registered [ 575.447450] Key type .llcrypt registered [ 575.536813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 575.543980] alg: No test for adler32 (adler32-zlib) [ 576.374406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 576.388747] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 580.435710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.146@tcp [ 585.232296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.251039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 585.262539] LNet: Added LNI 192.168.201.46@tcp1 [8/256/0/180] [ 585.949474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_interrupt -r 1 [ 585.969412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp1 -m GET -e local_interrupt -r 1 [ 585.986123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_interrupt -r 1 [ 586.000374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -m GET -e local_interrupt -r 1 [ 586.018458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp -m GET -e local_interrupt -r 1 [ 586.036347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp1 -m GET -e local_interrupt -r 1 [ 586.053812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -m GET -e local_interrupt -r 1 [ 586.072207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -m GET -e local_interrupt -r 1 [ 586.091437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 586.099452] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 586.106262] LNet: 1 local NIs in recovery (showing 1): 192.168.201.46@tcp [ 586.196685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 586.212316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 586.229093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 586.242926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 586.257726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 586.272848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 586.290939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 586.311251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 586.320818] LNet: 60042:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 586.332685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 586.354629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 586.372036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 586.390178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 586.409819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 586.427560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 586.445285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 586.658835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_dropped -r 1 [ 586.677321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp1 -m GET -e local_dropped -r 1 [ 586.700147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_dropped -r 1 [ 586.716750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -m GET -e local_dropped -r 1 [ 586.733987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp -m GET -e local_dropped -r 1 [ 586.760813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp1 -m GET -e local_dropped -r 1 [ 586.782473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -m GET -e local_dropped -r 1 [ 586.809811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -m GET -e local_dropped -r 1 [ 586.830913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 586.846767] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 586.852305] LNet: Skipped 2 previous similar messages [ 586.957957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 586.973645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 586.988388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 587.001378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 587.018605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 587.032822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 587.048149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 587.062719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 587.070124] LNet: 60163:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 587.077460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 587.090409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 587.105458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 587.121874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 587.138903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 587.158828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 587.176605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 587.402124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_aborted -r 1 [ 587.416359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp1 -m GET -e local_aborted -r 1 [ 587.435343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_aborted -r 1 [ 587.454125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -m GET -e local_aborted -r 1 [ 587.470673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp -m GET -e local_aborted -r 1 [ 587.487136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp1 -m GET -e local_aborted -r 1 [ 587.504984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -m GET -e local_aborted -r 1 [ 587.523605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -m GET -e local_aborted -r 1 [ 587.544256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 587.668424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 587.685365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 587.701563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 587.716949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 587.732284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 587.746414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 587.760277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 587.776655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 587.796395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 587.812412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 587.825954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 587.840698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 587.856542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 587.875568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 587.892590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 588.172933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_no_route -r 1 [ 588.190544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp1 -m GET -e local_no_route -r 1 [ 588.210438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_no_route -r 1 [ 588.231147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -m GET -e local_no_route -r 1 [ 588.250628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp -m GET -e local_no_route -r 1 [ 588.267951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp1 -m GET -e local_no_route -r 1 [ 588.287364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -m GET -e local_no_route -r 1 [ 588.309161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -m GET -e local_no_route -r 1 [ 588.328896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 588.342915] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 588.346935] LNet: Skipped 6 previous similar messages [ 588.448758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 588.464894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 588.481588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 588.505189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 588.519838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 588.535613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 588.551144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 588.567476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 588.577748] LNet: 60407:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 588.581598] LNet: 60407:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 588.595634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 588.611679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 588.629641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 588.648647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 588.667889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 588.692936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 588.716461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 588.985306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_timeout -r 1 [ 589.000910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp1 -m GET -e local_timeout -r 1 [ 589.018566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_timeout -r 1 [ 589.039799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -m GET -e local_timeout -r 1 [ 589.058775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp -m GET -e local_timeout -r 1 [ 589.075979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp1 -m GET -e local_timeout -r 1 [ 589.105895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -m GET -e local_timeout -r 1 [ 589.132382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -m GET -e local_timeout -r 1 [ 589.158418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 589.295642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 589.314309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 589.332882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 589.348285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 589.365129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 589.381742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 589.395568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 589.422192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 589.462803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 589.481402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 589.500945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 589.518500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 589.539459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 589.564600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 589.593638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 589.850755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_error -r 1 [ 589.869223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp1 -m GET -e local_error -r 1 [ 589.894023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_error -r 1 [ 589.920466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -m GET -e local_error -r 1 [ 589.939232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp -m GET -e local_error -r 1 [ 589.954348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp1 -m GET -e local_error -r 1 [ 589.970377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -m GET -e local_error -r 1 [ 589.990952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -m GET -e local_error -r 1 [ 590.014392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 590.133143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 590.147200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 590.164262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 590.180292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 590.199075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 590.220651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 590.238569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 590.269351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 590.286202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 590.310537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 590.331094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 590.360300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 590.384788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 590.400528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 590.418727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 593.236054] LNet: 60724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.239537] LNetError: 60724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 593.243088] LNet: Removed LNI 192.168.201.46@tcp [ 594.277891] LNet: Removed LNI 192.168.201.46@tcp1 [ 594.489306] Key type .llcrypt unregistered [ 594.491464] Key type ._llcrypt unregistered [ 601.926369] Key type ._llcrypt registered [ 601.928273] Key type .llcrypt registered [ 602.569993] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 14:11:40 (1784225500) [ 602.969311] Key type .llcrypt unregistered [ 602.970702] Key type ._llcrypt unregistered [ 606.011594] Key type ._llcrypt registered [ 606.013155] Key type .llcrypt registered [ 606.101104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 606.106698] alg: No test for adler32 (adler32-zlib) [ 606.938192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 606.969629] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 611.276375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.146@tcp [ 611.700157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e remote_dropped -r 1 [ 611.737240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e remote_dropped -r 1 [ 611.769938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 611.788470] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 611.944136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 611.960651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 611.984226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 612.013396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 612.035484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 612.048576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 612.064905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 612.079101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 612.086462] LNet: 61970:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 612.096205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 612.111386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 612.126890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 612.146357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 612.161744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 612.175330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 612.190359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 612.397595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e remote_error -r 1 [ 612.415228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e remote_error -r 1 [ 612.432577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 612.446113] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 612.565447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 612.580808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 612.599742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 612.614759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 612.628740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 612.644643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 612.659165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 612.676464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 612.686989] LNet: 62079:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 612.702779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 612.721675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 612.738803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 612.756024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 612.774076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 612.790574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 612.817627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 613.252940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e remote_timeout -r 1 [ 613.285992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e remote_timeout -r 1 [ 613.330565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 613.552393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 613.577291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 613.611687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 613.674819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 613.719760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 613.750456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 613.775657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 613.795627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 613.810250] LNet: 62188:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 613.831617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 613.867282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 613.887735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 613.904608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 613.922143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 613.939994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 613.956828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 614.133781] LNet: 62229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.136793] LNetError: 62229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.148960] LNet: Removed LNI 192.168.201.46@tcp [ 614.393471] Key type .llcrypt unregistered [ 614.395243] Key type ._llcrypt unregistered [ 633.041106] Key type ._llcrypt registered [ 633.044454] Key type .llcrypt registered [ 634.723797] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 14:12:11 (1784225531) [ 635.608496] Key type .llcrypt unregistered [ 635.614904] Key type ._llcrypt unregistered [ 642.573182] Key type ._llcrypt registered [ 642.575784] Key type .llcrypt registered [ 642.794974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 642.810297] alg: No test for adler32 (adler32-zlib) [ 643.792703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 643.826728] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 651.945933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.146@tcp [ 664.409482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.459614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 664.497472] LNet: Added LNI 192.168.201.46@tcp1 [8/256/0/180] [ 666.168203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e remote_dropped -r 1 [ 666.193534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp1 -m GET -e remote_dropped -r 1 [ 666.254210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e remote_dropped -r 1 [ 666.296313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -m GET -e remote_dropped -r 1 [ 666.330086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp -m GET -e remote_dropped -r 1 [ 666.397190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp1 -m GET -e remote_dropped -r 1 [ 666.458471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -m GET -e remote_dropped -r 1 [ 666.515576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -m GET -e remote_dropped -r 1 [ 666.579641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 666.610125] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 666.615577] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.146@tcp [ 666.831333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 666.850701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 666.872198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 666.905309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 666.958868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 666.996705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 667.041196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 667.079917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 667.108628] LNet: 63698:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 667.138222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 667.198197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 667.249867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 667.274896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 667.329216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 667.389066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 667.458865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 667.522832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 668.093921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e remote_error -r 1 [ 668.122462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp1 -m GET -e remote_error -r 1 [ 668.148205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e remote_error -r 1 [ 668.167824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -m GET -e remote_error -r 1 [ 668.201210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp -m GET -e remote_error -r 1 [ 668.227835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp1 -m GET -e remote_error -r 1 [ 668.254515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -m GET -e remote_error -r 1 [ 668.287952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -m GET -e remote_error -r 1 [ 668.312072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 668.321106] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 668.324431] LNet: Skipped 2 previous similar messages [ 668.552346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 668.621069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 668.683516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 668.728966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 668.772420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 668.815133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 668.841693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 668.886984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 668.921801] LNet: 63821:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 668.945436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 668.977565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 669.000549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 669.045766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 669.078349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 669.125083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 669.164371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 669.212606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 669.862985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e remote_timeout -r 1 [ 669.908366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp1 -m GET -e remote_timeout -r 1 [ 669.968277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e remote_timeout -r 1 [ 670.006405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -m GET -e remote_timeout -r 1 [ 670.066858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp -m GET -e remote_timeout -r 1 [ 670.145719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp1 -m GET -e remote_timeout -r 1 [ 670.201803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -m GET -e remote_timeout -r 1 [ 670.270705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -m GET -e remote_timeout -r 1 [ 670.327817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.146@tcp [ 670.344929] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 670.622618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 670.697189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 670.724702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 670.755798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 670.818669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 670.848758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 670.874027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 670.898023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 670.916672] LNet: 63944:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 670.946486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 670.988895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 671.077561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 671.108153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 671.148673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 671.210389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 671.243353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 671.302975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 675.472690] LNet: 64022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.478852] LNetError: 64022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 675.485343] LNet: Removed LNI 192.168.201.46@tcp [ 675.495925] LNet: Removed LNI 192.168.201.46@tcp1 [ 675.849359] Key type .llcrypt unregistered [ 675.851385] Key type ._llcrypt unregistered [ 691.049924] Key type ._llcrypt registered [ 691.052985] Key type .llcrypt registered [ 692.348220] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 14:13:09 (1784225589) [ 694.075079] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 695.836483] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 14:13:12 (1784225592) [ 696.574428] Key type .llcrypt unregistered [ 696.577948] Key type ._llcrypt unregistered [ 701.429127] Key type ._llcrypt registered [ 701.431658] Key type .llcrypt registered [ 701.563600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 701.587886] alg: No test for adler32 (adler32-zlib) [ 702.487709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 702.516778] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 709.508725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.146@tcp [ 710.122062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e network_timeout -r 1 [ 710.143898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e network_timeout -r 1 [ 710.163048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.146@tcp [ 710.177234] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 710.362686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 710.396290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 710.435969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 710.485912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 710.522827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 710.615547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 710.654783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 710.678795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 710.702097] LNet: 65567:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 710.729537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 710.756142] LNet: 1 local NIs in recovery (showing 1): 192.168.201.46@tcp [ 710.780330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 710.822948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 710.866819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 710.909038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 710.938957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 711.177267] LNet: 65606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.185922] LNetError: 65606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.206699] LNet: Removed LNI 192.168.201.46@tcp [ 711.571663] Key type .llcrypt unregistered [ 711.577233] Key type ._llcrypt unregistered [ 724.889610] Key type ._llcrypt registered [ 724.891442] Key type .llcrypt registered [ 725.078837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 725.093929] alg: No test for adler32 (adler32-zlib) [ 725.971540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 726.001531] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 731.286244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.146@tcp [ 736.783367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.828727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 736.851653] LNet: Added LNI 192.168.201.46@tcp1 [8/256/0/180] [ 737.962644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e network_timeout -r 1 [ 737.991383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp1 -m GET -e network_timeout -r 1 [ 738.010921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e network_timeout -r 1 [ 738.032881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -m GET -e network_timeout -r 1 [ 738.061421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp -m GET -e network_timeout -r 1 [ 738.088952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp1 -m GET -e network_timeout -r 1 [ 738.119555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -m GET -e network_timeout -r 1 [ 738.147944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -m GET -e network_timeout -r 1 [ 738.175836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.146@tcp [ 738.196124] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 738.275402] LNet: 1 local NIs in recovery (showing 1): 192.168.201.46@tcp [ 738.283214] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.146@tcp [ 738.440786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 738.455225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 738.491552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 738.525568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 738.560793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 738.583489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 738.601877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 738.621966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 738.636641] LNet: 66484:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 738.650916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 738.673138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 738.704963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 738.729938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 738.755714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 738.778950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 743.119561] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 743.351676] LNet: 66558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.357684] LNetError: 66558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.364919] LNet: Removed LNI 192.168.201.46@tcp [ 743.373977] LNet: Removed LNI 192.168.201.46@tcp1 [ 743.667613] Key type .llcrypt unregistered [ 743.670232] Key type ._llcrypt unregistered [ 755.885647] Key type ._llcrypt registered [ 755.886836] Key type .llcrypt registered [ 756.873340] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 14:14:14 (1784225654) [ 757.274794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 757.285389] alg: No test for adler32 (adler32-zlib) [ 758.200205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.266623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 758.291290] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 758.773449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 758.793020] LNet: Added LNI 192.168.201.46@tcp1 [8/256/0/180] [ 759.537566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 759.589733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 759.618724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 759.640698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 759.664666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 759.692638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 759.734741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.46@tcp [ 759.750921] LNetError: 67660:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.201.46@tcp to recovery [ 760.287309] LNet: 1 local NIs in recovery (showing 1): 192.168.201.46@tcp [ 760.293573] LNet: There was an unexpected network error while writing to 192.168.201.46: rc = -22 [ 762.337479] LNet: There was an unexpected network error while writing to 192.168.201.46: rc = -22 [ 763.181211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 766.432602] LNet: There was an unexpected network error while writing to 192.168.201.46: rc = -22 [ 766.600838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 766.662435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 766.726951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.742747] LNet: 67767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.747894] LNetError: 67767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 766.752776] LNet: Removed LNI 192.168.201.46@tcp [ 766.763835] LNet: Removed LNI 192.168.201.46@tcp1 [ 766.776422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.816759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 766.826937] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 767.277350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 767.291333] LNet: Added LNI 192.168.201.46@tcp1 [8/256/0/180] [ 767.888034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 767.922519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 767.965401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 768.001494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 768.041574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 768.074608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 768.102189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 768.128485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.46@tcp [ 768.144607] LNetError: 68039:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.201.46@tcp to recovery [ 768.864305] LNet: There was an unexpected network error while writing to 192.168.201.46: rc = -22 [ 771.604861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 775.014139] LNet: There was an unexpected network error while writing to 192.168.201.46: rc = -22 [ 775.017185] LNet: Skipped 1 previous similar message [ 779.369794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 779.405263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 779.428580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 784.676938] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 14:14:42 (1784225682) [ 784.744331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.777562] LNet: 68749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.791020] LNetError: 68749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.798775] LNetError: 68749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 784.804297] LNet: Removed LNI 192.168.201.46@tcp [ 784.820144] LNet: Removed LNI 192.168.201.46@tcp1 [ 784.856229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.918612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 784.936555] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 785.572465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 785.593851] LNet: Added LNI 192.168.201.46@tcp1 [8/256/0/180] [ 786.257599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 786.310502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 786.359287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 786.401903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 786.438697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 786.469957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 786.500609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.46@tcp --health 0 [ 786.599577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 786.975369] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.46@tcp [ 787.999985] LNet: There was an unexpected network error while writing to 192.168.201.46: rc = -22 [ 788.008102] LNet: Skipped 1 previous similar message [ 801.803238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 801.948164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 801.976185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 802.006366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.46@tcp --health 500 [ 802.076395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 808.694601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 808.759829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.775715] LNet: 69247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.780753] LNetError: 69247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.785576] LNetError: 69247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 808.790601] LNet: Removed LNI 192.168.201.46@tcp [ 808.797981] LNet: Removed LNI 192.168.201.46@tcp1 [ 808.815917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.861653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 808.880053] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 809.574928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 809.588537] LNet: Added LNI 192.168.201.46@tcp1 [8/256/0/180] [ 810.334108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 810.355627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 810.441591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 810.478098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 810.514905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 810.541459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 810.570754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 810.600812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.46@tcp --health 0 [ 811.936692] LNet: There was an unexpected network error while writing to 192.168.201.46: rc = -22 [ 811.944953] LNet: Skipped 3 previous similar messages [ 813.000783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 822.096809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 822.137860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 822.172720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 827.344222] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 14:15:24 (1784225724) [ 827.642647] LNet: 70293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.646968] LNetError: 70293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.650695] LNetError: 70293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 827.657555] LNet: Removed LNI 192.168.201.46@tcp [ 827.671067] LNet: Removed LNI 192.168.201.46@tcp1 [ 828.118159] Key type .llcrypt unregistered [ 828.122664] Key type ._llcrypt unregistered [ 832.957328] Key type ._llcrypt registered [ 832.959332] Key type .llcrypt registered [ 833.109269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 833.119937] alg: No test for adler32 (adler32-zlib) [ 834.000697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 834.030972] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 840.221182] Lustre: DEBUG MARKER: Initial discovery [ 840.378843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.146@tcp [ 841.891247] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 842.062898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 842.096429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.146@tcp [ 842.112725] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 842.124727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 842.162293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.146@tcp -r 1 -m GET -l 3 [ 842.207280] LNet: 1 local NIs in recovery (showing 1): 192.168.201.46@tcp [ 842.214224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.146@tcp -r 1 -m GET -e local_error [ 842.243783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.146@tcp -r 1 -m PUT -l 6 [ 843.192415] Lustre: DEBUG MARKER: Force oleg146-client.virtnet to discover 192.168.201.146@tcp (in background) [ 843.495158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.146@tcp [ 844.576292] Lustre: DEBUG MARKER: Force oleg146-server to discover 192.168.201.46@tcp [ 846.559326] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 846.701290] Lustre: DEBUG MARKER: Wait for 71069 [ 848.038507] Lustre: DEBUG MARKER: Finished wait on 71069 [ 848.135505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 848.178451] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 848.202254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 848.554236] LNet: 71518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.558335] LNetError: 71518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.567201] LNet: Removed LNI 192.168.201.46@tcp [ 848.891234] Key type .llcrypt unregistered [ 848.894438] Key type ._llcrypt unregistered [ 862.415643] Key type ._llcrypt registered [ 862.416989] Key type .llcrypt registered [ 863.305961] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 14:16:00 (1784225760) [ 863.659207] Key type .llcrypt unregistered [ 863.660593] Key type ._llcrypt unregistered [ 867.915210] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 868.077774] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 868.080564] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 868.083715] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 870.512730] Key type ._llcrypt registered [ 870.515764] Key type .llcrypt registered [ 870.711834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 870.729732] alg: No test for adler32 (adler32-zlib) [ 871.609848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.650052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 871.674411] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 872.221602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 872.239105] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 878.377452] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 14:16:15 (1784225775) [ 878.610566] LNet: 73588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.623716] LNetError: 73588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 878.629611] LNet: Removed LNI 10.1.2.3@tcp [ 878.636522] LNet: Removed LNI 192.168.201.46@tcp [ 879.083651] Key type .llcrypt unregistered [ 879.086867] Key type ._llcrypt unregistered [ 883.661710] Key type ._llcrypt registered [ 883.664246] Key type .llcrypt registered [ 883.828538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 883.840498] alg: No test for adler32 (adler32-zlib) [ 884.756414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.809585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 884.844137] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 885.377990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 885.404087] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 886.280531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.46@tcp [ 887.327191] LNetError: Unexpected error -22 connecting to 192.168.201.46@tcp at host 192.168.201.46:988 [ 890.367625] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 14:16:28 (1784225788) [ 890.481383] LNet: 74714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.486205] LNetError: 74714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 890.491240] LNet: Removed LNI 10.1.2.3@tcp [ 891.566387] LNet: Removed LNI 192.168.201.46@tcp [ 891.962757] Key type .llcrypt unregistered [ 891.963709] Key type ._llcrypt unregistered [ 896.410270] Key type ._llcrypt registered [ 896.412530] Key type .llcrypt registered [ 896.547943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 896.561657] alg: No test for adler32 (adler32-zlib) [ 897.399849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.432299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 897.447421] LNet: Added LNI 192.168.201.46@tcp1 [8/256/0/180] [ 897.952812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 897.980559] LNet: Added LNI 192.168.201.46@tcp2 [8/256/0/180] [ 898.552828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.46@tcp1 --nid 192.168.201.46@tcp2 [ 901.228365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.46@tcp1 192.168.201.46@tcp2 [ 901.255123] LNetError: 75559:0:(lib-move.c:2063:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.46@tcp1 to 192.168.201.46@tcp2 - Route table may be misconfigured [ 901.270976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.46@tcp2 192.168.201.46@tcp1 [ 904.855869] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 14:16:42 (1784225802) [ 904.972797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.999445] LNet: 76132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.006309] LNetError: 76132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.014264] LNet: Removed LNI 192.168.201.46@tcp1 [ 906.088078] LNet: Removed LNI 192.168.201.46@tcp2 [ 906.114667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.161744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 906.185542] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 906.677815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 906.694490] LNet: Added LNI 192.168.201.46@tcp1 [8/256/0/180] [ 907.311889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 907.336475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.46@tcp 192.168.201.46@tcp [ 907.358815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.46@tcp1 192.168.201.46@tcp1 [ 907.388831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.46@tcp -d 192.168.201.46@tcp -e network_timeout [ 907.422317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -e network_timeout [ 907.452209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -e network_timeout [ 907.482798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -e network_timeout [ 907.508481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 907.524516] LNet: There was an unexpected network error while writing to 192.168.201.46: rc = -22 [ 907.538385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 907.598518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 907.684170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 908.191263] LNet: 1 local NIs in recovery (showing 1): 192.168.201.46@tcp [ 911.793780] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 14:16:49 (1784225809) [ 911.996455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.008930] LNet: 76995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.011946] LNetError: 76995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.015679] LNetError: 76995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 912.018806] LNet: Removed LNI 192.168.201.46@tcp [ 913.062078] LNet: Removed LNI 192.168.201.46@tcp1 [ 913.077953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.144667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 913.351714] LNet: 77032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.587578] Key type .llcrypt unregistered [ 913.588984] Key type ._llcrypt unregistered [ 920.764480] Key type ._llcrypt registered [ 920.768166] Key type .llcrypt registered [ 921.728736] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 14:16:59 (1784225819) [ 921.984720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 922.006421] alg: No test for adler32 (adler32-zlib) [ 922.922314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.081502] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 925.473592] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 14:17:02 (1784225822) [ 925.751874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 925.792703] LNet: 78129:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.822466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.925508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 925.971407] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 926.761884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 926.781651] LNet: Added LNI 192.168.201.46@tcp1 [8/256/0/180] [ 927.278149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 927.311269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp1 [ 927.330635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp1 [ 931.849639] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 14:17:09 (1784225829) [ 933.255819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg146-server [ 934.602933] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 14:17:12 (1784225832) [ 935.971952] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg146-server [ 937.003414] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 14:17:14 (1784225834) [ 938.265864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg146-server [ 939.252158] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 14:17:17 (1784225837) [ 940.387013] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg146-server [ 941.403346] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 14:17:19 (1784225839) [ 942.594790] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg146-server [ 943.764818] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 14:17:21 (1784225841) [ 945.097939] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg146-server [ 946.129481] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 14:17:23 (1784225843) [ 947.276565] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg146-server [ 948.379616] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 14:17:26 (1784225846) [ 949.442469] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg146-server [ 950.722929] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 14:17:28 (1784225848) [ 952.278799] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg146-server [ 953.627469] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 14:17:31 (1784225851) [ 953.664180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.676926] LNet: 81999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.681713] LNetError: 81999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.686525] LNet: Removed LNI 192.168.201.46@tcp [ 954.734401] LNet: Removed LNI 192.168.201.46@tcp1 [ 954.753424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.815499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 954.863249] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 955.410936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 955.474402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 956.744861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.761412] LNet: 82161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.766537] LNetError: 82161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.771202] LNetError: 82161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 956.786613] LNet: Removed LNI 192.168.201.46@tcp [ 956.809852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.851457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 956.864671] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 957.390345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 957.507295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.571944] LNet: Removed LNI 192.168.201.46@tcp [ 958.591869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.634448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 958.662565] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 959.186622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 959.233451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 959.325283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 959.342311] LNet: 82453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.348152] LNet: 82453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 959.353506] LNetError: 82453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.357806] LNetError: 82453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 959.372761] LNet: Removed LNI 192.168.201.46@tcp [ 959.391121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.435860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 959.460109] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 959.986647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 960.096440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.124927] LNet: Removed LNI 192.168.201.46@tcp [ 961.138839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.177288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 961.201752] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 961.692254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 961.753531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 961.848954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.860643] LNetError: 82745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.864709] LNetError: 82745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 961.874601] LNet: Removed LNI 192.168.201.46@tcp [ 961.886176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.916448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 961.937722] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 962.402882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 962.535185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.557824] LNet: Removed LNI 192.168.201.46@tcp [ 963.569987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.599387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 963.613389] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 964.003565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 964.056186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 964.143833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 964.153483] LNet: 83038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.155958] LNet: 83038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 965.229738] LNet: Removed LNI 192.168.201.46@tcp [ 965.248256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.287967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 965.301524] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 965.744599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 965.872945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.890230] LNetError: 83183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.893460] LNetError: 83183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 965.903199] LNet: Removed LNI 192.168.201.46@tcp [ 965.915866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.942291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 965.956488] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 966.402946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 966.442899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 966.554418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.589083] LNet: Removed LNI 192.168.201.46@tcp [ 967.606623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.649198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 967.669708] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 968.257335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 968.444404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.510464] LNet: Removed LNI 192.168.201.46@tcp [ 969.538906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.585982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 969.601911] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 970.114946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 970.154893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 970.312357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.368150] LNet: Removed LNI 192.168.201.46@tcp [ 971.393559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.441299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 971.461600] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 972.090355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 972.272922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 972.295759] LNet: 83768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.300287] LNet: 83768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 973.349533] LNet: Removed LNI 192.168.201.46@tcp [ 973.363982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.404837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 973.427800] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 973.951842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 974.001122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.021752] LNetError: 83900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.026951] LNetError: 83900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 974.039564] LNet: Removed LNI 192.168.201.46@tcp [ 974.060738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.144608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 974.165416] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 974.701773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 974.745496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.762685] LNet: Removed LNI 192.168.201.46@tcp [ 974.776299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.828789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 974.878173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.900489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.932258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 974.990964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 975.014240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.052559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 975.093583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 975.118744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.152801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 975.179673] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 975.667458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 975.704278] LNet: Removed LNI 192.168.201.46@tcp [ 975.726261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.779148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 980.068869] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 14:17:57 (1784225877) [ 980.119137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.151683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.238501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 980.256339] LNet: Added LNI 192.168.201.46@tcp231 [8/256/0/180] [ 981.048644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.073938] LNet: Removed LNI 192.168.201.46@tcp231 [ 981.088245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.147771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 981.165633] LNet: Added LNI 192.168.201.46@tcp231 [8/256/0/180] [ 981.169435] LNet: 84934:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 981.431422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 981.467086] LNet: Removed LNI 192.168.201.46@tcp231 [ 981.485513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 981.504326] LNet: Added LNI 192.168.201.46@tcp231 [8/256/0/180] [ 982.200162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.220352] LNet: Removed LNI 192.168.201.46@tcp231 [ 982.234097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.289601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 982.326759] LNet: Added LNI 192.168.201.46@tcp231 [8/256/0/0] [ 982.331794] LNet: 85086:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 982.544342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 982.569685] LNet: Removed LNI 192.168.201.46@tcp231 [ 982.577761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 982.597027] LNet: Added LNI 192.168.201.46@tcp231 [8/256/0/0] [ 983.412120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.439529] LNet: Removed LNI 192.168.201.46@tcp231 [ 983.458467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.524268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 983.542881] LNet: Added LNI 192.168.201.46@tcp231 [8/256/0/60] [ 983.548985] LNet: 85237:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 983.820177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 983.839904] LNet: Removed LNI 192.168.201.46@tcp231 [ 983.850461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 983.866932] LNet: Added LNI 192.168.201.46@tcp231 [8/256/0/60] [ 984.653892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 984.680186] LNet: Removed LNI 192.168.201.46@tcp231 [ 984.690094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.736521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 984.753098] LNet: Added LNI 192.168.201.46@tcp231 [8/256/0/180] [ 984.994717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 985.022525] LNet: Removed LNI 192.168.201.46@tcp231 [ 985.041933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 985.068815] LNet: Added LNI 192.168.201.46@tcp231 [8/256/0/180] [ 990.594912] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 14:18:08 (1784225888) [ 990.931816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 990.957342] LNet: 86098:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.964550] LNet: 86098:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 990.979915] LNetError: 86098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.982857] LNetError: 86098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 991.013340] LNet: Removed LNI 192.168.201.46@tcp231 [ 991.041305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.145931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 991.183405] LNet: Added LNI 192.168.201.46@tcp232 [8/256/0/180] [ 991.672047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 991.689580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 991.716497] LNet: Removed LNI 192.168.201.46@tcp232 [ 991.728678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 991.745576] LNet: Added LNI 192.168.201.46@tcp232 [8/256/0/180] [ 992.342667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 992.372554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 992.400691] LNet: Removed LNI 192.168.201.46@tcp232 [ 992.412887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 992.423369] LNet: Added LNI 192.168.201.46@tcp232 [8/256/0/180] [ 992.893801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 992.918746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 992.943771] LNet: Removed LNI 192.168.201.46@tcp232 [ 992.954829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 992.987327] LNet: Added LNI 192.168.201.46@tcp232 [8/256/0/180] [ 993.471441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 993.492626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 993.516977] LNet: Removed LNI 192.168.201.46@tcp232 [ 993.526411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 997.085076] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 14:18:14 (1784225894) [ 997.748535] Key type .llcrypt unregistered [ 997.751196] Key type ._llcrypt unregistered [ 1002.479226] Key type ._llcrypt registered [ 1002.482330] Key type .llcrypt registered [ 1002.650076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1002.659462] alg: No test for adler32 (adler32-zlib) [ 1003.519391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1003.551598] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1010.121343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.146@tcp [ 1017.891684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.019691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1018.048209] LNet: Added LNI 192.168.201.46@tcp1 [8/256/0/180] [ 1019.760665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_interrupt -r 2 [ 1019.813260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp1 -m GET -e local_interrupt -r 2 [ 1019.874457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_interrupt -r 2 [ 1019.947349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -m GET -e local_interrupt -r 2 [ 1020.007993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp -m GET -e local_interrupt -r 2 [ 1020.048181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp1 -m GET -e local_interrupt -r 2 [ 1020.083824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -m GET -e local_interrupt -r 2 [ 1020.136045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -m GET -e local_interrupt -r 2 [ 1020.217635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.46@tcp 192.168.201.146@tcp [ 1020.241616] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 1020.252390] LNet: 1 local NIs in recovery (showing 1): 192.168.201.46@tcp [ 1020.540676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1020.565647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1020.599933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1020.626673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1020.655906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1020.688936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1020.731690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1020.763816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1020.787048] LNet: 87887:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1020.819155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1020.864178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1020.936478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1020.999229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1021.074457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1021.132827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1022.075890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_dropped -r 2 [ 1022.113215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp1 -m GET -e local_dropped -r 2 [ 1022.184710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_dropped -r 2 [ 1022.233900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -m GET -e local_dropped -r 2 [ 1022.315781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp -m GET -e local_dropped -r 2 [ 1022.357456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp1 -m GET -e local_dropped -r 2 [ 1022.400175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -m GET -e local_dropped -r 2 [ 1022.479095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -m GET -e local_dropped -r 2 [ 1022.529222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.46@tcp 192.168.201.146@tcp [ 1022.559244] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 1022.914563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1022.942367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1022.970399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1022.993304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1023.013159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1023.034160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1023.054837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1023.072555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1023.083152] LNet: 88008:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1023.095184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1023.117109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1023.142231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1023.161241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1023.184197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1023.205299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1023.573188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_aborted -r 2 [ 1023.618810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp1 -m GET -e local_aborted -r 2 [ 1023.645982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_aborted -r 2 [ 1023.679169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -m GET -e local_aborted -r 2 [ 1023.708475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp -m GET -e local_aborted -r 2 [ 1023.743438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp1 -m GET -e local_aborted -r 2 [ 1023.775936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -m GET -e local_aborted -r 2 [ 1023.813230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -m GET -e local_aborted -r 2 [ 1023.865675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.46@tcp 192.168.201.146@tcp [ 1023.888887] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 1024.255652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1024.291151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1024.339360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1024.373751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1024.402837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1024.431269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1024.464313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1024.495510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1024.531940] LNet: 88128:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1024.566860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1024.618974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1024.663956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1024.702911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1024.735466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1024.764618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1025.272675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_no_route -r 2 [ 1025.333575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp1 -m GET -e local_no_route -r 2 [ 1025.365881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_no_route -r 2 [ 1025.400740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -m GET -e local_no_route -r 2 [ 1025.434631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp -m GET -e local_no_route -r 2 [ 1025.461368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp1 -m GET -e local_no_route -r 2 [ 1025.488957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -m GET -e local_no_route -r 2 [ 1025.518959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -m GET -e local_no_route -r 2 [ 1025.548504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.46@tcp 192.168.201.146@tcp [ 1025.776314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1025.796808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1025.818678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1025.841590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1025.871547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1025.901368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1025.934105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1025.961583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1025.991053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1026.025857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1026.047779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1026.074595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1026.100842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1026.132502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1026.603712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e local_timeout -r 2 [ 1026.646112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp1 -m GET -e local_timeout -r 2 [ 1026.687196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e local_timeout -r 2 [ 1026.716695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -m GET -e local_timeout -r 2 [ 1026.743726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp -m GET -e local_timeout -r 2 [ 1026.767360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp1 -m GET -e local_timeout -r 2 [ 1026.802484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -m GET -e local_timeout -r 2 [ 1026.828790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -m GET -e local_timeout -r 2 [ 1026.852599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.46@tcp 192.168.201.146@tcp [ 1026.867193] LNet: There was an unexpected network error while writing to 192.168.201.146: rc = -22 [ 1026.871512] LNet: Skipped 1 previous similar message [ 1027.063236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1027.103460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1027.145536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1027.193529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1027.251427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1027.281955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1027.314341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1027.357510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1027.377635] LNet: 88368:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1027.382984] LNet: 88368:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1027.400453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1027.432392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1027.463375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1027.499558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1027.534312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1027.562950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1028.069961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp -m GET -e remote_dropped -r 2 [ 1028.103297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.146@tcp1 -m GET -e remote_dropped -r 2 [ 1028.147268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp -m GET -e remote_dropped -r 2 [ 1028.178191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp -d 192.168.201.46@tcp1 -m GET -e remote_dropped -r 2 [ 1028.210423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp -m GET -e remote_dropped -r 2 [ 1028.242176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.146@tcp1 -m GET -e remote_dropped -r 2 [ 1028.275752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp -m GET -e remote_dropped -r 2 [ 1028.335471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.46@tcp1 -d 192.168.201.46@tcp1 -m GET -e remote_dropped -r 2 [ 1028.372156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.46@tcp 192.168.201.146@tcp [ 1028.387023] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.146@tcp [ 1028.644771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1028.668559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1028.701568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1028.730868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1028.768425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1028.797233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1028.822818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1028.843663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1028.874925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1028.909671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1028.971338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1029.042494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1029.069886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1029.135633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1033.792599] LNet: 88562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1033.796957] LNetError: 88562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1033.801825] LNet: Removed LNI 192.168.201.46@tcp [ 1033.817645] LNet: Removed LNI 192.168.201.46@tcp1 [ 1034.289545] Key type .llcrypt unregistered [ 1034.293367] Key type ._llcrypt unregistered [ 1048.714573] Key type ._llcrypt registered [ 1048.721391] Key type .llcrypt registered [ 1049.625722] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 14:19:07 (1784225947) [ 1050.012693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1050.025938] alg: No test for adler32 (adler32-zlib) [ 1050.905389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1050.947307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1050.968572] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1051.751523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1051.785732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.46@tcp [ 1051.802290] LNetError: 89555:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.201.46@tcp to recovery [ 1051.869855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.46@tcp [ 1051.963955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1056.502564] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 14:19:13 (1784225953) [ 1056.676433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1056.687076] LNet: 90145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.691372] LNetError: 90145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1056.701795] LNet: Removed LNI 192.168.201.46@tcp [ 1056.727605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1056.780922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1056.803515] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1057.261808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1057.283471] LNet: Added LNI 192.168.201.46@tcp1 [8/256/0/180] [ 1057.976905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.46@tcp [ 1057.993394] LNetError: 90411:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.201.46@tcp to recovery [ 1058.847985] LNet: 1 local NIs in recovery (showing 1): 192.168.201.46@tcp [ 1058.864580] LNet: There was an unexpected network error while writing to 192.168.201.46: rc = -22 [ 1059.196513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.46@tcp [ 1066.418295] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 14:19:23 (1784225963) [ 1067.694192] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg146-server [ 1068.926580] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 14:19:26 (1784225966) [ 1069.944284] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg146-server [ 1071.238226] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 14:19:28 (1784225968) [ 1072.631952] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg146-server [ 1074.172484] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 14:19:31 (1784225971) [ 1074.371461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1074.387727] LNet: 92039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.399777] LNetError: 92039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1074.407741] LNet: Removed LNI 192.168.201.46@tcp [ 1074.425473] LNet: Removed LNI 192.168.201.46@tcp1 [ 1074.456312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1074.519536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1074.549074] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1075.129811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1075.190609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1075.224335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1075.285955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1075.316598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1075.356898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1079.630306] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 14:19:37 (1784225977) [ 1079.767813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1079.799518] LNet: 92764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1079.803499] LNetError: 92764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1079.808951] LNetError: 92764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1079.820194] LNet: Removed LNI 192.168.201.46@tcp [ 1079.834616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1079.871747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1079.884851] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1085.075287] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 14:19:42 (1784225982) [ 1085.294887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1085.309979] LNet: 93468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.314284] LNetError: 93468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1085.334287] LNet: Removed LNI 192.168.201.46@tcp [ 1085.352161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1085.393669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1085.411447] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1091.146517] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 14:19:48 (1784225988) [ 1092.158756] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1093.445624] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 14:19:50 (1784225990) [ 1093.726422] LNet: 94503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1093.729376] LNetError: 94503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1093.739987] LNet: Removed LNI 192.168.201.46@tcp [ 1094.167604] Key type .llcrypt unregistered [ 1094.170607] Key type ._llcrypt unregistered [ 1098.264308] Key type ._llcrypt registered [ 1098.265903] Key type .llcrypt registered [ 1098.399236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1098.409244] alg: No test for adler32 (adler32-zlib) [ 1099.299352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1099.326948] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1104.897088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.146@tcp [ 1108.571742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.146@tcp [ 1108.803987] LNet: 94920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1108.808729] LNetError: 94920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1108.823463] LNet: Removed LNI 192.168.201.46@tcp [ 1109.118391] Key type .llcrypt unregistered [ 1109.121153] Key type ._llcrypt unregistered [ 1116.306526] Key type ._llcrypt registered [ 1116.310412] Key type .llcrypt registered [ 1117.237199] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1118.305826] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1119.602707] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 14:20:17 (1784226017) [ 1120.834632] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1122.091089] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 14:20:19 (1784226019) [ 1123.475498] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg146-server [ 1124.788347] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 14:20:22 (1784226022) [ 1126.102514] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg146-server [ 1127.267431] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 14:20:24 (1784226024) [ 1128.558975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1128.572746] alg: No test for adler32 (adler32-zlib) [ 1129.613495] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 1129.745664] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1131.367194] Key type lgssc registered [ 1132.254167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1145.659547] Key type lgssc unregistered [ 1145.846753] LNet: 98092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.853676] LNetError: 98092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1145.869165] LNet: Removed LNI 192.168.201.46@tcp [ 1146.351714] Key type .llcrypt unregistered [ 1146.356583] Key type ._llcrypt unregistered [ 1156.984199] Key type ._llcrypt registered [ 1156.985807] Key type .llcrypt registered [ 1158.036701] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 14:20:55 (1784226055) [ 1158.407236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1158.419208] alg: No test for adler32 (adler32-zlib) [ 1159.284958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1159.321302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1159.352322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1159.387377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1159.419355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1163.586887] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 14:21:01 (1784226061) [ 1163.787908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1163.804183] LNet: 99506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.821316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1163.868505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1163.886019] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1164.521845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1164.547563] LNet: 99650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1164.555959] LNetError: 99650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1164.571596] LNet: Removed LNI 192.168.201.46@tcp [ 1164.582754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1164.626734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1164.645194] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1164.916528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1164.946316] LNet: Removed LNI 192.168.201.46@tcp [ 1164.963066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1165.001987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1165.014823] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1169.836529] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 14:21:07 (1784226067) [ 1169.905273] LNet: 100262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1169.912284] LNet: 100262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1169.926029] LNetError: 100262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1169.934939] LNetError: 100262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1169.948531] LNet: Removed LNI 192.168.201.46@tcp [ 1170.489796] Key type .llcrypt unregistered [ 1170.492645] Key type ._llcrypt unregistered [ 1175.358493] Key type ._llcrypt registered [ 1175.360514] Key type .llcrypt registered [ 1175.503282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1175.525209] alg: No test for adler32 (adler32-zlib) [ 1176.430047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1176.496859] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1176.498507] LNet: *** cfs_fail_loc=e003, val=100*** [ 1176.985627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 1176.999327] LNet: *** cfs_fail_loc=e003, val=100*** [ 1177.001428] LNet: Skipped 2 previous similar messages [ 1177.006879] Ping reply received with mapping: [ 1177.008913] found metadata reply NI[192.168.201.46@tcp] [ 1177.010977] 2e c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1177.013420] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1177.015916] 5a 5a 5a 5a 5a 5a 5a 5a [ 1177.088081] LNet: 100666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.096287] LNetError: 100666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1178.152748] LNet: Removed LNI 192.168.201.46@tcp [ 1178.612184] Key type .llcrypt unregistered [ 1178.613515] Key type ._llcrypt unregistered [ 1186.053991] Key type ._llcrypt registered [ 1186.055699] Key type .llcrypt registered [ 1186.969593] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 14:21:24 (1784226084) [ 1187.385290] Key type .llcrypt unregistered [ 1187.388394] Key type ._llcrypt unregistered [ 1192.028834] Key type ._llcrypt registered [ 1192.030840] Key type .llcrypt registered [ 1192.168502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1192.178592] alg: No test for adler32 (adler32-zlib) [ 1193.178507] LNetError: 101710:0:(api-ni.c:2795:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1193.397070] Key type .llcrypt unregistered [ 1193.398922] Key type ._llcrypt unregistered [ 1197.078267] Key type ._llcrypt registered [ 1197.081329] Key type .llcrypt registered [ 1198.087820] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 14:21:35 (1784226095) [ 1198.673602] Key type .llcrypt unregistered [ 1198.676586] Key type ._llcrypt unregistered [ 1203.850862] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1203.854946] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1206.297844] Key type ._llcrypt registered [ 1206.300312] Key type .llcrypt registered [ 1206.404294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1206.411776] alg: No test for adler32 (adler32-zlib) [ 1207.292089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1207.338540] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1215.534988] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 14:21:52 (1784226112) [ 1215.600555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1215.615969] LNet: 103484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.623568] LNetError: 103484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1215.640072] LNet: Removed LNI 10.1.2.3@tcp [ 1215.674723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1216.266137] Key type .llcrypt unregistered [ 1216.267391] Key type ._llcrypt unregistered [ 1220.523411] Key type ._llcrypt registered [ 1220.527755] Key type .llcrypt registered [ 1220.678491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1220.706787] alg: No test for adler32 (adler32-zlib) [ 1221.624922] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1221.706697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 1221.834212] LNet: 103778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.838927] LNetError: 103778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1222.889319] LNet: Removed LNI 192.168.201.46@tcp [ 1223.308574] Key type .llcrypt unregistered [ 1223.309951] Key type ._llcrypt unregistered [ 1231.377172] Key type ._llcrypt registered [ 1231.378852] Key type .llcrypt registered [ 1232.372307] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 14:22:09 (1784226129) [ 1232.846259] Key type .llcrypt unregistered [ 1232.849689] Key type ._llcrypt unregistered [ 1237.222506] Key type ._llcrypt registered [ 1237.225315] Key type .llcrypt registered [ 1237.400783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1237.429777] alg: No test for adler32 (adler32-zlib) [ 1242.427781] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 14:22:20 (1784226140) [ 1242.577640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1242.602390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1242.657903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1242.676103] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1243.204906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1243.230604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1243.254597] LNet: Removed LNI 192.168.201.46@tcp [ 1243.435686] LNet: 105521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.748525] Key type .llcrypt unregistered [ 1243.750758] Key type ._llcrypt unregistered [ 1251.062373] Key type ._llcrypt registered [ 1251.063804] Key type .llcrypt registered [ 1251.930958] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 14:22:29 (1784226149) [ 1252.329488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1252.342163] alg: No test for adler32 (adler32-zlib) [ 1253.251547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1253.301365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1253.325444] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1253.861655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 1253.893861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.46@tcp [ 1253.927524] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1253.930912] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1253.933987] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1253, 0, 0 [ 1253.941038] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1253.944133] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1253.948327] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1253.951950] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1253, 0, 0 [ 1253.956342] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1253.958727] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1253.961378] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1253.966453] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1253, 0, 1303 [ 1253.974398] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1253.977635] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1253.985517] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1253.987788] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1253, 0, 1303 [ 1253.993503] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1258.736603] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 14:22:36 (1784226156) [ 1259.115518] LNet: 107059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1259.124818] LNetError: 107059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1259.136216] LNet: Removed LNI 192.168.201.46@tcp [ 1259.620971] Key type .llcrypt unregistered [ 1259.627103] Key type ._llcrypt unregistered [ 1264.414579] Key type ._llcrypt registered [ 1264.416349] Key type .llcrypt registered [ 1264.552816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1264.562917] alg: No test for adler32 (adler32-zlib) [ 1265.446653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1265.472596] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1271.087651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.146@tcp [ 1277.075890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1277.108752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1277.133321] LNet: Added LNI 192.168.201.46@tcp1 [8/256/0/180] [ 1277.807174] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1277.993857] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1278.000474] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1278.003572] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1280.250368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1280.268938] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1287.799180] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 14:23:05 (1784226185) [ 1287.897360] LNet: 108560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1287.903536] LNetError: 108560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1287.911904] LNet: Removed LNI 192.168.201.46@tcp [ 1288.927183] LNet: Removed LNI 192.168.201.46@tcp1 [ 1288.929568] LNetError: 108560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1288.934282] LNetError: 108560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1288.950388] LNet: Removed LNI 10.1.2.3@tcp99 [ 1289.502753] Key type .llcrypt unregistered [ 1289.505541] Key type ._llcrypt unregistered [ 1293.588791] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1293.591635] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1296.058877] Key type ._llcrypt registered [ 1296.063887] Key type .llcrypt registered [ 1296.219711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1296.236047] alg: No test for adler32 (adler32-zlib) [ 1297.113226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1297.144516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1297.162883] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1297.676985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1297.692668] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1299.051992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1299.092889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1299.126708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1299.138667] LNetError: 109310:0:(peer.c:2276:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1299.168983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1299.210631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1299.510712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1299.564903] LNet: 109330:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1299.571338] LNetError: 109330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1299.578028] LNet: Removed LNI 10.1.2.3@tcp [ 1300.657553] LNet: Removed LNI 192.168.201.46@tcp [ 1301.214472] Key type .llcrypt unregistered [ 1301.216564] Key type ._llcrypt unregistered [ 1308.967537] Key type ._llcrypt registered [ 1308.969523] Key type .llcrypt registered [ 1309.817464] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 14:23:27 (1784226207) [ 1310.181138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1310.190170] alg: No test for adler32 (adler32-zlib) [ 1311.075830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1311.116682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1311.140720] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1311.585970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.46@tcp [ 1311.621359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg146-client@tcp [ 1315.442868] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 14:23:33 (1784226213) [ 1315.537075] LNet: 110857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1315.542396] LNetError: 110857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1315.566625] LNet: Removed LNI 192.168.201.46@tcp [ 1316.081384] Key type .llcrypt unregistered [ 1316.083810] Key type ._llcrypt unregistered [ 1320.458618] Key type ._llcrypt registered [ 1320.460828] Key type .llcrypt registered [ 1320.619538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1320.633662] alg: No test for adler32 (adler32-zlib) [ 1321.488304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1321.503984] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1321.531775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1325.796857] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 14:23:43 (1784226223) [ 1326.938447] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1328.205111] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 14:23:45 (1784226225) [ 1329.392483] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1330.563980] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 14:23:48 (1784226228) [ 1330.755751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1330.771455] LNet: 112281:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1330.774654] LNetError: 112281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1330.795608] LNet: Removed LNI 192.168.201.46@tcp [ 1330.808536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1330.834615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1330.848992] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1331.356383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1331.397262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1335.443427] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 14:23:53 (1784226233) [ 1335.488288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1335.500581] LNet: 112984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.505506] LNetError: 112984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1335.516245] LNet: Removed LNI 192.168.201.46@tcp [ 1335.531718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1335.580890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1335.601225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1336.012358] Key type .llcrypt unregistered [ 1336.014683] Key type ._llcrypt unregistered [ 1343.888365] Key type ._llcrypt registered [ 1343.892179] Key type .llcrypt registered [ 1344.843294] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 14:24:02 (1784226242) [ 1345.229275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1345.239645] alg: No test for adler32 (adler32-zlib) [ 1346.116404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1346.149663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1346.169061] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1346.729292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1346.780909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1351.217286] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 14:24:08 (1784226248) [ 1351.340689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1351.358279] LNet: 114544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1351.364968] LNetError: 114544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1351.377923] LNet: Removed LNI 192.168.201.46@tcp [ 1351.394734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1351.429948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1351.450710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1355.507500] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 14:24:13 (1784226253) [ 1355.571642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1355.591235] LNet: 115119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1355.608986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1355.677388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1355.995823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1356.017412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1356.052567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1356.321691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1356.352325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1356.415529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1356.435399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1356.485995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1356.729957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1356.754990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1356.800353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1357.047175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1357.082672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1361.370727] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 14:24:18 (1784226258) [ 1361.604614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1361.621695] LNet: 115758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.624984] LNet: 115758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1361.644878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1361.696567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1361.717114] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1362.121187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1362.139495] LNet: Added LNI 192.168.201.46@tcp2 [8/256/0/180] [ 1362.684638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1362.726356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1362.901737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1362.924575] LNetError: 116037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1363.937242] LNet: Removed LNI 192.168.201.46@tcp [ 1363.943287] LNetError: 116037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1363.958625] LNet: Removed LNI 192.168.201.46@tcp2 [ 1363.976839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1364.042571] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1364.049682] LNet: Added LNI 192.168.201.46@tcp2 [8/256/0/180] [ 1364.055724] LNet: 116044:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1364.385300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1364.500412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1364.510952] LNet: 116081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1364.514707] LNet: 116081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1365.535683] LNet: Removed LNI 192.168.201.46@tcp [ 1365.552868] LNet: Removed LNI 192.168.201.46@tcp2 [ 1365.579207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1365.624730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1365.650148] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1366.124575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1366.145681] LNet: Added LNI 192.168.201.46@tcp2 [8/256/0/180] [ 1366.858385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1366.888956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1366.956781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1367.156692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1367.174226] LNetError: 116365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1367.180713] LNetError: 116365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1367.186411] LNet: Removed LNI 192.168.201.46@tcp [ 1368.230563] LNet: Removed LNI 192.168.201.46@tcp2 [ 1368.247301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1368.295430] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1368.299863] LNet: Added LNI 192.168.201.46@tcp2 [8/256/0/180] [ 1368.307335] LNet: 116372:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1368.737498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1368.889363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1368.919213] LNet: 116409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1368.922436] LNet: 116409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1368.928309] LNet: Removed LNI 192.168.201.46@tcp [ 1369.959745] LNet: Removed LNI 192.168.201.46@tcp2 [ 1369.975277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1370.016136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1370.039589] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1370.492986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1370.510431] LNet: Added LNI 192.168.201.46@tcp2 [8/256/0/180] [ 1371.157649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1371.204983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1371.322731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1371.332944] LNetError: 116688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.335645] LNetError: 116688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1371.338283] LNet: Removed LNI 192.168.201.46@tcp [ 1371.349652] LNet: Removed LNI 192.168.201.46@tcp2 [ 1371.359617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1371.387522] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1371.390961] LNet: Added LNI 192.168.201.46@tcp2 [8/256/0/180] [ 1371.393598] LNet: 116694:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1371.759974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1371.953505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1373.023587] LNet: Removed LNI 192.168.201.46@tcp [ 1373.031164] LNet: Removed LNI 192.168.201.46@tcp2 [ 1373.048432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1373.091419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1373.114159] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1373.574899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1373.591306] LNet: Added LNI 192.168.201.46@tcp2 [8/256/0/180] [ 1374.081017] hrtimer: interrupt took 4007763 ns [ 1374.227623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1374.257495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1374.300594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1374.449722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1375.519710] LNet: Removed LNI 192.168.201.46@tcp [ 1376.550755] LNet: Removed LNI 192.168.201.46@tcp2 [ 1376.577727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1376.629738] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1376.635719] LNet: Added LNI 192.168.201.46@tcp2 [8/256/0/180] [ 1376.639892] LNet: 117022:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1377.070544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1377.224456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1377.238302] LNet: 117059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.242894] LNet: 117059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1377.246664] LNet: Removed LNI 192.168.201.46@tcp [ 1378.282893] LNet: Removed LNI 192.168.201.46@tcp2 [ 1378.306976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1378.359593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1378.377909] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1378.905573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1378.923631] LNet: Added LNI 192.168.201.46@tcp2 [8/256/0/180] [ 1379.546067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1379.572461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1379.632117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1379.947394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1379.958638] LNetError: 117356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1379.961562] LNetError: 117356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1379.964652] LNet: Removed LNI 192.168.201.46@tcp [ 1379.970555] LNet: Removed LNI 192.168.201.46@tcp2 [ 1379.986963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1380.032136] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1380.036982] LNet: Added LNI 192.168.201.46@tcp2 [8/256/0/180] [ 1380.359343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1380.632589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1381.663181] LNet: Removed LNI 192.168.201.46@tcp [ 1381.678245] LNet: Removed LNI 192.168.201.46@tcp2 [ 1381.704391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.750635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1381.770671] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1382.319454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1382.338588] LNet: Added LNI 192.168.201.46@tcp2 [8/256/0/180] [ 1382.960249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1382.997963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1383.027189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1383.085174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1383.347927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1384.423691] LNet: Removed LNI 192.168.201.46@tcp [ 1385.447737] LNet: Removed LNI 192.168.201.46@tcp2 [ 1385.468789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1385.513678] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1385.519548] LNet: Added LNI 192.168.201.46@tcp2 [8/256/0/180] [ 1385.525561] LNet: 117722:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1385.533296] LNet: 117722:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1385.907893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1390.621782] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1391.785710] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 14:24:49 (1784226289) [ 1391.896995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1391.930112] LNet: Removed LNI 192.168.201.46@tcp [ 1392.999826] LNet: Removed LNI 192.168.201.46@tcp2 [ 1393.024379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1393.222995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1393.241880] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1393.792895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1393.803682] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1394.407186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1394.425934] LNet: 118756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1394.430986] LNet: 118756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1394.436766] LNet: Removed LNI 10.1.2.3@tcp [ 1394.447277] LNet: Removed LNI 192.168.201.46@tcp [ 1394.462232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1394.511190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1394.529362] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1394.532684] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1394.774475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1394.790926] LNet: Removed LNI 192.168.201.46@tcp [ 1394.803175] LNet: Removed LNI 10.1.2.3@tcp [ 1399.739118] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 14:24:57 (1784226297) [ 1399.814065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1399.877805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1400.025055] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1400.180716] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1400.182725] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1400.184855] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1402.453505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1402.481958] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1403.011535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1403.023013] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1404.579790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.46@tcp [ 1404.606765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1404.642775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.46@tcp -m PUT -e local_timeout -r 1 [ 1404.693506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1404.739848] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1404.742930] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1404.903140] LNetError: 119422:0:(lib-move.c:4771:lnet_parse()) 192.168.201.46@tcp, src 192.168.201.46@tcp: Dropping PUT (error -108 looking up sender) [ 1405.920487] LNetError: 119879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1405.928479] LNetError: 119879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 1405.932208] LNet: Removed LNI 10.1.2.3@tcp [ 1406.954648] LNet: Removed LNI 192.168.201.46@tcp [ 1407.234392] Key type .llcrypt unregistered [ 1407.237592] Key type ._llcrypt unregistered [ 1416.475575] Key type ._llcrypt registered [ 1416.478437] Key type .llcrypt registered [ 1417.559187] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 14:25:14 (1784226314) [ 1417.786187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1417.804210] alg: No test for adler32 (adler32-zlib) [ 1418.668903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1418.918783] LNet: 120731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1419.319527] Key type .llcrypt unregistered [ 1419.323647] Key type ._llcrypt unregistered [ 1423.972774] Key type ._llcrypt registered [ 1423.974574] Key type .llcrypt registered [ 1424.126702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1424.145073] alg: No test for adler32 (adler32-zlib) [ 1425.031054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1425.046028] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1430.653527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.146@tcp [ 1430.855502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1431.234225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.146@tcp [ 1431.424661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1431.452854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1431.480362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1431.509250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1431.557893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1431.581418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1431.615885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1431.642453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1431.657945] LNet: 121181:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1431.675704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1431.695299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1431.718530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1431.735808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1431.757814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1431.781461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1432.200622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1432.223315] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.146@tcp [ 1432.363093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1432.390282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1432.409661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1432.432856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1432.454333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1432.486408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1432.510793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1432.530585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1432.547403] LNet: 121298:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1432.568524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1432.594386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1432.621557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1432.646427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1432.675415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1432.695816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1433.155873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.146@tcp [ 1433.285913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1433.303296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1433.320655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1433.340607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1433.362447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1433.381207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1433.403972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1433.424493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1433.450403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1433.473625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1433.493630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1433.526503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1433.561989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1433.584537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1434.093685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1434.270724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1434.312580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1434.336788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1434.360663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1434.384802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1434.401391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1434.418238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1434.440394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1434.452480] LNet: 121532:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1434.455708] LNet: 121532:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1434.468611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1434.500486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1434.541465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1434.556933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1434.576840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1434.603941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1434.812285] LNet: 121578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1434.817765] LNetError: 121578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.837529] LNet: Removed LNI 192.168.201.46@tcp [ 1435.108929] Key type .llcrypt unregistered [ 1435.111125] Key type ._llcrypt unregistered [ 1447.110616] Key type ._llcrypt registered [ 1447.112730] Key type .llcrypt registered [ 1448.036863] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 14:25:45 (1784226345) [ 1448.271987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1448.282983] alg: No test for adler32 (adler32-zlib) [ 1449.163782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.445731] LNet: 122464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.871294] Key type .llcrypt unregistered [ 1449.872735] Key type ._llcrypt unregistered [ 1453.143937] Key type ._llcrypt registered [ 1453.145512] Key type .llcrypt registered [ 1453.258758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1453.269470] alg: No test for adler32 (adler32-zlib) [ 1454.158907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1454.185076] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1459.126964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.146@tcp [ 1465.306456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1465.348513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1465.368867] LNet: Added LNI 192.168.201.46@tcp1 [8/256/0/180] [ 1466.187309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1466.611683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.146@tcp [ 1466.820286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1466.845585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1466.861877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1466.882891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1466.914024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1466.942576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1466.965832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1466.985278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1466.994186] LNet: 123114:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1467.005601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1467.029734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1467.050753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1467.075253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1467.107480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1467.132932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1467.617194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1467.802153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1467.821755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1467.844344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1467.866465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1467.903124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1467.929214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1467.948750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1467.970209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1467.983400] LNet: 123232:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1468.003696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1468.038665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1468.072089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1468.101624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1468.127687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1468.155309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1468.609329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.146@tcp [ 1468.792454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1468.818668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1468.849383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1468.898087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1468.925369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1468.959360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1468.996878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1469.019128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1469.029530] LNet: 123349:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1469.039585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1469.081690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1469.114271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1469.142703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1469.169912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1469.203608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1469.666922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1469.794542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1469.808809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1469.823934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1469.840101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1469.865914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1469.889772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1469.905107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1469.921187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1469.936483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1469.959333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1469.983422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1470.019126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1470.046879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1470.078893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1474.009872] LNet: 123546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.014811] LNetError: 123546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1474.021791] LNet: Removed LNI 192.168.201.46@tcp [ 1474.030307] LNet: Removed LNI 192.168.201.46@tcp1 [ 1474.362831] Key type .llcrypt unregistered [ 1474.364362] Key type ._llcrypt unregistered [ 1485.081450] Key type ._llcrypt registered [ 1485.084133] Key type .llcrypt registered [ 1485.795830] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 14:26:23 (1784226383) [ 1486.896814] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 1488.057367] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 14:26:25 (1784226385) [ 1493.165856] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 14:26:30 (1784226390) [ 1497.058707] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 14:26:34 (1784226394) [ 1501.312615] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 14:26:38 (1784226398) [ 1505.396306] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 14:26:43 (1784226403) [ 1510.309275] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 14:26:47 (1784226407) [ 1515.092386] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 14:26:52 (1784226412) [ 1518.699330] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 14:26:56 (1784226416) [ 1522.855072] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 14:27:00 (1784226420) [ 1526.386501] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 14:27:04 (1784226424) [ 1530.565529] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 14:27:08 (1784226428) [ 1534.972633] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 14:27:12 (1784226432) [ 1539.116375] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 14:27:16 (1784226436) [ 1542.944214] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 14:27:20 (1784226440) [ 1547.492586] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 14:27:25 (1784226445) [ 1552.336512] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 14:27:29 (1784226449) [ 1557.232539] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 14:27:34 (1784226454) [ 1561.915830] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 14:27:39 (1784226459) [ 1562.200525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1562.214236] alg: No test for adler32 (adler32-zlib) [ 1563.065636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.147979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1563.166816] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1563.831730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1563.844841] LNet: 134537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.848369] LNetError: 134537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1563.857559] LNet: Removed LNI 192.168.201.46@tcp [ 1563.868071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.908980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1563.929809] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1567.753975] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 14:27:45 (1784226465) [ 1572.218845] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 14:27:49 (1784226469) [ 1576.861457] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 14:27:54 (1784226474) [ 1581.300827] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 14:27:58 (1784226478) [ 1585.034706] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 14:28:02 (1784226482) [ 1589.239984] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 14:28:06 (1784226486) [ 1593.253646] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 14:28:10 (1784226490) [ 1596.660196] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 14:28:14 (1784226494) [ 1600.533532] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 14:28:18 (1784226498) [ 1605.171938] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 14:28:22 (1784226502) [ 1608.850642] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 14:28:26 (1784226506) [ 1613.242277] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 14:28:30 (1784226510) [ 1617.263354] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1468 sec ========== 14:28:34 (1784226514) [ 1617.420578] LNet: 141996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.426653] LNetError: 141996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.437934] LNet: Removed LNI 192.168.201.46@tcp [ 1617.829629] Key type .llcrypt unregistered [ 1617.831964] Key type ._llcrypt unregistered [ 1625.220346] Key type ._llcrypt registered [ 1625.222046] Key type .llcrypt registered [ 1625.490144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1625.499728] alg: No test for adler32 (adler32-zlib) [ 1626.512894] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 1626.671458] LNet: Added LNI 192.168.201.46@tcp [8/256/0/180] [ 1628.303167] Key type lgssc registered [ 1629.196306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1668.093891] Lustre: Mounted lustre-client [ 1671.644966] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1676.229651] Lustre: Unmounted lustre-client [ 1709.379373] Key type lgssc unregistered [ 1709.550787] LNet: 144921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1709.557389] LNetError: 144921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.567565] LNet: Removed LNI 192.168.201.46@tcp [ 1709.994199] Key type .llcrypt unregistered [ 1709.996895] Key type ._llcrypt unregistered