[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 432477281 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002404] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007361] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010143] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013083] *** VALIDATE selinux *** [ 0.021320] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025847] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026173] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029039] *** VALIDATE tmpfs *** [ 0.030471] *** VALIDATE proc *** [ 0.031275] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.034112] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036065] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.041944] debug: unmapping init [mem 0xffffffff8f259000-0xffffffff8f260fff] [ 0.043164] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044681] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047015] ... generic registers: 4 [ 0.048012] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051009] ... event mask: 000000070000000f [ 0.052317] rcu: Hierarchical SRCU implementation. [ 0.054482] smp: Bringing up secondary CPUs ... [ 0.055481] x86: Booting SMP configuration: [ 0.056020] .... node #0, CPUs: #1 #2 #3 [ 0.059304] smp: Brought up 1 node, 4 CPUs [ 0.061012] smpboot: Max logical packages: 1 [ 0.062019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147391] node 0 deferred pages initialised in 83ms [ 0.150109] devtmpfs: initialized [ 0.151214] x86/mm: Memory block size: 128MB [ 0.153707] gcov: version magic: 0x41383552 [ 0.155298] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159112] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161316] pinctrl core: initialized pinctrl subsystem [ 0.163152] [ 0.163725] ************************************************************* [ 0.166013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170014] ** ** [ 0.172011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174011] ** ** [ 0.177012] ** This means that this kernel is built to expose internal ** [ 0.179010] ** IOMMU data structures, which may compromise security on ** [ 0.181013] ** your system. ** [ 0.183012] ** ** [ 0.186014] ** If you see this message and you are not debugging the ** [ 0.188011] ** kernel, report this immediately to your vendor! ** [ 0.190011] ** ** [ 0.192013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195012] ************************************************************* [ 0.197622] NET: Registered protocol family 16 [ 0.199445] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.203070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.207063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211011] cpuidle: using governor menu [ 0.213077] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.215397] PCI: Using configuration type 1 for base access [ 0.218127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.226091] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.229022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.232140] cryptd: max_cpu_qlen set to 1000 [ 0.236223] ACPI: Added _OSI(Module Device) [ 0.238017] ACPI: Added _OSI(Processor Device) [ 0.239008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.241013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.245692] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.251582] ACPI: Interpreter enabled [ 0.253057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.255013] ACPI: Using IOAPIC for interrupt routing [ 0.257096] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.261375] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.271511] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.274042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.277019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.281079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.286431] acpiphp: Slot [2] registered [ 0.288157] acpiphp: Slot [5] registered [ 0.290168] acpiphp: Slot [6] registered [ 0.292208] acpiphp: Slot [3] registered [ 0.294097] acpiphp: Slot [4] registered [ 0.295107] acpiphp: Slot [7] registered [ 0.297136] acpiphp: Slot [8] registered [ 0.299110] acpiphp: Slot [9] registered [ 0.301120] acpiphp: Slot [10] registered [ 0.302178] acpiphp: Slot [11] registered [ 0.304106] acpiphp: Slot [12] registered [ 0.305103] acpiphp: Slot [13] registered [ 0.307136] acpiphp: Slot [14] registered [ 0.309067] acpiphp: Slot [15] registered [ 0.310142] acpiphp: Slot [16] registered [ 0.311086] acpiphp: Slot [17] registered [ 0.313098] acpiphp: Slot [18] registered [ 0.314106] acpiphp: Slot [19] registered [ 0.316107] acpiphp: Slot [20] registered [ 0.318197] acpiphp: Slot [21] registered [ 0.319096] acpiphp: Slot [22] registered [ 0.321103] acpiphp: Slot [23] registered [ 0.323123] acpiphp: Slot [24] registered [ 0.325134] acpiphp: Slot [25] registered [ 0.326131] acpiphp: Slot [26] registered [ 0.328104] acpiphp: Slot [27] registered [ 0.330291] acpiphp: Slot [28] registered [ 0.331067] acpiphp: Slot [29] registered [ 0.332150] acpiphp: Slot [30] registered [ 0.334157] acpiphp: Slot [31] registered [ 0.336066] PCI host bridge to bus 0000:00 [ 0.338022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.340027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.343026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.347024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.350026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.352031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.354175] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.357333] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.360381] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.367544] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.372056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.376021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.379019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.382087] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.384605] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.387577] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.390048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.393944] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.398031] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.409745] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.414019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.420341] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.427013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.431031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.444014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.453913] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.459018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.466023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.477025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.487464] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.490506] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.493393] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.496402] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.498203] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.504143] iommu: Default domain type: Passthrough [ 0.508290] SCSI subsystem initialized [ 0.509166] ACPI: bus type USB registered [ 0.511178] usbcore: registered new interface driver usbfs [ 0.513116] usbcore: registered new interface driver hub [ 0.515115] usbcore: registered new device driver usb [ 0.517239] pps_core: LinuxPPS API ver. 1 registered [ 0.519022] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.522133] PTP clock support registered [ 0.525072] EDAC MC: Ver: 3.0.0 [ 0.526520] PCI: Using ACPI for IRQ routing [ 0.527651] NetLabel: Initializing [ 0.528013] NetLabel: domain hash size = 128 [ 0.529008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.530126] NetLabel: unlabeled traffic allowed by default [ 0.532162] vgaarb: loaded [ 0.533276] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.536017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.543372] clocksource: Switched to clocksource kvm-clock [ 0.647739] VFS: Disk quotas dquot_6.6.0 [ 0.649065] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.650953] *** VALIDATE ramfs *** [ 0.652448] *** VALIDATE hugetlbfs *** [ 0.654342] pnp: PnP ACPI init [ 0.657045] pnp: PnP ACPI: found 6 devices [ 0.678283] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.681621] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.683796] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.686322] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.688679] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.691016] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.693279] NET: Registered protocol family 2 [ 0.696062] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.701274] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.704320] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.709456] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.713397] TCP: Hash tables configured (established 65536 bind 65536) [ 0.716453] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.719602] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.722617] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.724833] NET: Registered protocol family 1 [ 0.727458] RPC: Registered named UNIX socket transport module. [ 0.730144] RPC: Registered udp transport module. [ 0.732171] RPC: Registered tcp transport module. [ 0.734219] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.737123] NET: Registered protocol family 44 [ 0.738900] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.741478] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.743422] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.745482] PCI: CLS 0 bytes, default 64 [ 0.746827] Unpacking initramfs... [ 2.149113] debug: unmapping init [mem 0xffff935fbcc64000-0xffff935fbffcffff] [ 2.154756] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.156821] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.159442] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.672847] Initialise system trusted keyrings [ 2.674806] Key type blacklist registered [ 2.677077] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.687327] zbud: loaded [ 2.690920] *** VALIDATE nfs *** [ 2.692496] *** VALIDATE nfs4 *** [ 2.694518] pstore: using deflate compression [ 2.698571] Platform Keyring initialized [ 2.799757] NET: Registered protocol family 38 [ 2.801800] Key type asymmetric registered [ 2.803339] Asymmetric key parser 'x509' registered [ 2.805468] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.808839] io scheduler mq-deadline registered [ 2.810649] io scheduler kyber registered [ 2.812419] io scheduler bfq registered [ 2.814230] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.816402] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.819705] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.822949] ACPI: Power Button [PWRF] [ 2.828598] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.834942] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.842262] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.868162] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.899573] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.905743] Non-volatile memory driver v1.3 [ 2.908359] Linux agpgart interface v0.103 [ 2.938738] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.942421] vda: detected capacity change from 0 to 76779520 [ 2.956527] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.960396] vdb: detected capacity change from 0 to 1073741824 [ 2.970840] libphy: Fixed MDIO Bus: probed [ 2.976137] usbcore: registered new interface driver usbserial_generic [ 2.978840] usbserial: USB Serial support registered for generic [ 2.981296] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.985413] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.987146] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.989398] mousedev: PS/2 mouse device common for all mice [ 2.992286] rtc_cmos 00:05: RTC can wake from S4 [ 2.994587] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.995588] rtc_cmos 00:05: registered as rtc0 [ 3.000405] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.001882] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.004348] intel_pstate: CPU model not supported [ 3.010142] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.014733] hid: raw HID events driver (C) Jiri Kosina [ 3.017368] usbcore: registered new interface driver usbhid [ 3.019808] usbhid: USB HID core driver [ 3.021517] drop_monitor: Initializing network drop monitor service [ 3.024355] Initializing XFRM netlink socket [ 3.026698] NET: Registered protocol family 10 [ 3.030361] Segment Routing with IPv6 [ 3.032081] NET: Registered protocol family 17 [ 3.034320] mpls_gso: MPLS GSO support [ 3.042919] RAS: Correctable Errors collector initialized. [ 3.045397] AVX version of gcm_enc/dec engaged. [ 3.047257] AES CTR mode by8 optimization enabled [ 3.127648] sched_clock: Marking stable (3127617026, 0)->(4066097518, -938480492) [ 3.131402] registered taskstats version 1 [ 3.133390] Loading compiled-in X.509 certificates [ 3.136224] zswap: loaded using pool lzo/zbud [ 3.162240] Key type big_key registered [ 3.175051] Key type encrypted registered [ 3.176781] ima: No TPM chip found, activating TPM-bypass! [ 3.179276] ima: Allocated hash algorithm: sha1 [ 3.180771] ima: No architecture policies found [ 3.182491] evm: Initialising EVM extended attributes: [ 3.184194] evm: security.selinux [ 3.185311] evm: security.ima [ 3.186334] evm: security.capability [ 3.187574] evm: HMAC attrs: 0x1 [ 3.189933] rtc_cmos 00:05: setting system clock to 2026-09-16 13:29:59 UTC (1789565399) [ 3.196276] debug: unmapping init [mem 0xffffffff90203000-0xffffffff903fffff] [ 3.199432] debug: unmapping init [mem 0xffffffff8ef82000-0xffffffff8f258fff] [ 3.206128] Write protecting the kernel read-only data: 28672k [ 3.209696] debug: unmapping init [mem 0xffffffff8d603000-0xffffffff8d7fffff] [ 3.211954] debug: unmapping init [mem 0xffffffff8df14000-0xffffffff8dffffff] [ 3.244298] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.251480] systemd[1]: Detected virtualization kvm. [ 3.253113] systemd[1]: Detected architecture x86-64. [ 3.254921] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.280052] systemd[1]: No hostname configured. [ 3.281825] systemd[1]: Set hostname to . [ 3.283413] random: systemd: uninitialized urandom read (16 bytes read) [ 3.286110] systemd[1]: Initializing machine ID from random generator. [ 3.335740] random: ln: uninitialized urandom read (6 bytes read) [ 3.425570] random: systemd: uninitialized urandom read (16 bytes read) [ 3.428458] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.433665] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.439833] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Timers. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.122789] device-mapper: uevent: version 1.0.3 [ 4.125378] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [[ 4.547222] random: fast init done  OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.025991] virtio_net virtio0 ens2: renamed from eth0 [ 5.057867] scsi host0: ata_piix [ 5.077391] scsi host1: ata_piix [ 5.079036] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.081522] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.498905] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.693345] random: crng init done [ 9.694586] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.197335] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.387418] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.648196] SELinux: Disabled at runtime. [ 11.710213] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.719943] systemd[1]: Detected virtualization kvm. [ 11.721954] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.215954] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.219413] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.224858] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.230320] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.232673] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.238702] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.244710] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 12.370980] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.669655] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.990605] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.028396] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.103878] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.119549] EDAC sbridge: Ver: 1.1.2 [ 14.116045] Key type dns_resolver registered [ 14.418611] NFS: Registering the id_resolver key type [ 14.420700] Key type id_resolver registered [ 14.422439] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg333-client login: [ 46.837783] libcfs: loading out-of-tree module taints kernel. [ 46.915204] Key type ._llcrypt registered [ 46.922056] Key type .llcrypt registered [ 47.252973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 47.260342] alg: No test for adler32 (adler32-zlib) [ 48.309065] Lustre: Lustre: Build Version: 2.17.58_40_g81e4dae [ 48.709307] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 50.359183] Key type lgssc registered [ 51.465310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 55.540012] hrtimer: interrupt took 2955969 ns [ 197.348107] Lustre: Mounted lustre-client - version 2.17.58_40_g81e4dae [ 203.318669] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 220.020477] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing check_logdir /tmp/testlogs/ [ 223.199220] Lustre: lustre-OST0000-osc-ffff936009b72800: disconnect after 23s idle [ 226.112812] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing yml_node [ 231.462314] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 234.124934] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 236.999731] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 238.996263] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Sep 16 09:33:53 EDT 2026 [ 257.160943] Lustre: Unmounted lustre-client [ 286.359196] Key type lgssc unregistered [ 286.683165] LNet: 8427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 286.700044] LNetError: 8427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 286.721517] LNet: Removed LNI 192.168.203.33@tcp [ 287.497274] Key type .llcrypt unregistered [ 287.500171] Key type ._llcrypt unregistered [ 315.752506] Key type ._llcrypt registered [ 315.754218] Key type .llcrypt registered [ 316.020441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 316.043200] alg: No test for adler32 (adler32-zlib) [ 317.095423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 317.182570] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 317.217263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 321.730372] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 321.831952] LNet: 9296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.837074] LNetError: 9296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 321.861663] LNet: Removed LNI 192.168.203.33@tcp [ 322.550443] Key type .llcrypt unregistered [ 322.553348] Key type ._llcrypt unregistered [ 329.329527] Key type ._llcrypt registered [ 329.335422] Key type .llcrypt registered [ 331.186856] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:35:25 (1789565725) [ 331.664537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 331.702262] alg: No test for adler32 (adler32-zlib) [ 332.700788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.813724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.835056] LNet: 9712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.851699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.914723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 332.946991] LNet: 9719:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 340.101807] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:35:34 (1789565734) [ 340.285996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.342244] LNet: 10293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.404342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.576730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 347.916973] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:35:42 (1789565742) [ 348.131600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.207219] LNet: 10885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.243582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.380578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 355.468992] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:35:50 (1789565750) [ 355.628237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.672996] LNet: 11464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.710837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.854682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 364.451898] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:35:58 (1789565758) [ 364.622692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.682768] LNet: 12048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.714758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.852359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 365.495364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 373.192734] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 09:36:08 (1789565768) [ 373.341741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.373831] LNet: 12638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.418527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.527111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 374.210214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 374.535272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 382.351701] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 09:36:17 (1789565777) [ 382.534807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.626736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.746180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 393.470899] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 09:36:27 (1789565787) [ 393.533533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.558726] LNet: 13821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.565078] LNet: 13821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 393.583065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.627047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 393.660786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 393.923315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 393.948597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 394.318754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 394.345828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 394.935545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 394.999564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 395.781150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 395.814643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 396.472789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 396.545045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 397.355847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 397.442260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 398.142465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 398.185350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 398.592518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 398.636177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 398.981546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 399.022302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 399.240717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 399.263266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 399.594880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 399.659227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 399.965368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 400.060618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 408.389845] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 09:36:42 (1789565802) [ 408.456296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.505782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.616995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 408.646739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 417.353927] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 09:36:51 (1789565811) [ 417.450580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.487912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.598077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 417.704350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 425.363157] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 09:37:00 (1789565820) [ 425.633313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.660658] LNet: 15682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.667094] LNet: 15682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 425.683945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.781209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 425.831936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 433.605374] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 09:37:08 (1789565828) [ 433.728178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.791410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.903589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 433.958633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 441.559204] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 09:37:16 (1789565836) [ 441.894054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.938637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.079422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 442.151842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 450.278925] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 09:37:24 (1789565844) [ 450.394063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.449009] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.572824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 450.616774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 458.187084] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 09:37:32 (1789565852) [ 458.308344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.388520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.547103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 458.996936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 466.924549] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 09:37:41 (1789565861) [ 467.103882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.219631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.533485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 468.209786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 477.224915] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 09:37:51 (1789565871) [ 477.371640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.473814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.734559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 478.078318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 486.779305] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 09:38:01 (1789565881) [ 486.876852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.955481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.530551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 487.852978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 496.590758] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 09:38:11 (1789565891) [ 496.766429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.834466] LNet: 20509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 496.843635] LNet: 20509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 496.891949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.058337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 497.117031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 505.239233] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 09:38:19 (1789565899) [ 505.390970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.459249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.568048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 505.645746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 515.316592] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 09:38:29 (1789565909) [ 515.543036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.646314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.907856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 523.970986] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 09:38:38 (1789565918) [ 524.422352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.493590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.583887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 534.032226] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 09:38:48 (1789565928) [ 534.118900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.190798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.339595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 535.115258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 543.159053] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 09:38:57 (1789565937) [ 543.511928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.587095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.733548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 543.770978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 553.117827] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 09:39:07 (1789565947) [ 553.346039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.478180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.609791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 553.666471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 562.021398] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 09:39:16 (1789565956) [ 562.241924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.346544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.472150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 562.507498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 572.705395] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 09:39:27 (1789565967) [ 572.960043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.001083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.187898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 573.252433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 583.039492] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 09:39:37 (1789565977) [ 583.567977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.642644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.814700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 583.931835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 583.966983] LNetError: 25834:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 584.049358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 584.103143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 593.940159] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 09:39:48 (1789565988) [ 594.504725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.628090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.827369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.xLUTebSgho [ 594.915548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.xLUTebSgho [ 594.961474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.xLUTebSgho [ 602.580892] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 09:39:57 (1789565997) [ 602.749327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.813760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.877500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 602.920150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 602.981829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 609.407618] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 09:40:04 (1789566004) [ 609.782547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.883278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.001719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 610.034097] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 611.824771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 613.657844] Lustre: Lustre: Build Version: 2.17.58_40_g81e4dae [ 615.648072] Key type lgssc registered [ 617.208619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 622.146206] LNetError: 27596:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.203.33@tcp99 has no available nets [ 626.043686] Key type lgssc unregistered [ 626.586886] LNet: 28702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.605029] LNet: 28702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 626.617901] LNetError: 28702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 626.643948] LNet: Removed LNI 192.168.203.33@tcp [ 627.921638] Key type .llcrypt unregistered [ 627.927813] Key type ._llcrypt unregistered [ 638.858588] Key type ._llcrypt registered [ 638.862682] Key type .llcrypt registered [ 640.988380] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 09:40:35 (1789566035) [ 641.947544] Key type .llcrypt unregistered [ 641.953970] Key type ._llcrypt unregistered [ 650.402373] Key type ._llcrypt registered [ 650.411857] Key type .llcrypt registered [ 650.835734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 650.869281] alg: No test for adler32 (adler32-zlib) [ 651.956565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.105701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 652.152130] LNet: 29612:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 652.176776] LNet: 29612:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 652.195136] LNet: 29612:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 652.217782] LNet: Message forwarding will be enabled in 2s [ 655.332471] LNet: Message forwarding enabled [ 655.920302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 655.958159] LNet: Message forwarding will stop in 2s to 6s [ 659.433376] LNet: Message forwarding disabled [ 659.574372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 659.624273] LNet: 29702:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 659.637180] LNet: 29702:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 659.649596] LNet: 29702:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 659.667264] LNet: Message forwarding will be enabled in 2s [ 659.873839] LNet: 29715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.497448] Key type .llcrypt unregistered [ 660.500572] Key type ._llcrypt unregistered [ 673.846355] Key type ._llcrypt registered [ 673.857410] Key type .llcrypt registered [ 675.914519] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 09:41:09 (1789566069) [ 676.445101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 676.463354] alg: No test for adler32 (adler32-zlib) [ 677.418199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.518193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 685.932456] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 09:41:20 (1789566080) [ 686.062527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.103672] LNet: 31107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.139402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.229449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 686.286258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 686.327856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 686.409855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 686.484844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 686.581447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 686.648900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 686.716342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 686.736034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 686.763806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 686.800930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 686.829097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 686.867840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 686.905433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 686.963221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 687.022311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 687.062329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 687.110928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 687.149562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 687.163422] LNetError: 31157:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 695.647327] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 09:41:30 (1789566090) [ 695.874860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.953732] LNet: 31723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.026414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.165260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 705.250771] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 09:41:39 (1789566099) [ 708.037620] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg333-server [ 710.670832] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 09:41:44 (1789566104) [ 710.809305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.866317] LNet: 32657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.930525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.053602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 711.084689] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 712.245614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[34-38]@tcp [ 712.300110] LNet: 32795:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 712.366474] LNetError: 32666:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.34@tcp is being used as a gateway but routing feature is not turned on [ 712.382789] LNetError: 32666:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.34@tcp has gone from up to down [ 720.839758] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 09:41:55 (1789566115) [ 723.191954] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg333-server [ 725.049788] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 09:41:59 (1789566119) [ 725.408201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.443635] LNet: 33732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.453781] LNetError: 33732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.479064] LNet: Removed LNI 192.168.203.33@tcp [ 725.547146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.635158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 725.690721] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 726.799875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[34-39/2]@tcp [ 726.815199] LNet: 33850:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 726.824076] LNet: 33850:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 726.847376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[34-39/2]@tcp [ 734.781080] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 09:42:09 (1789566129) [ 734.902115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.953616] LNet: 34437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.965734] LNetError: 34437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 735.001580] LNet: Removed LNI 192.168.203.33@tcp [ 735.030164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.173709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 735.230913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.293129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.420438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 735.495368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.542135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.619171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 735.773785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.805707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.891875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 736.028653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.082468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.162709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 736.254690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.289173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.366720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 736.461933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.508717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.606602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 736.733721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.778696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.860708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 736.965302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.048771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.118688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 737.220303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.262590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.338526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 737.418546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.459968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.542356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 737.636468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.713886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.868138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 745.694587] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 09:42:20 (1789566140) [ 748.091875] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg333-server [ 750.066763] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 09:42:24 (1789566144) [ 752.524472] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg333-server [ 754.838456] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 09:42:29 (1789566149) [ 754.989506] LNet: 35836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.001058] LNet: 35836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 755.744367] Key type .llcrypt unregistered [ 755.749387] Key type ._llcrypt unregistered [ 763.452518] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 763.760855] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 763.777145] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 763.795260] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 766.506233] Key type ._llcrypt registered [ 766.516264] Key type .llcrypt registered [ 766.862341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 766.896752] alg: No test for adler32 (adler32-zlib) [ 767.918208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 767.987866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 768.024582] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 768.743723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 768.781402] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 771.955530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 771.985384] LNet: Removed LNI 10.1.2.3@tcp [ 772.145496] LNet: 36563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.158312] LNetError: 36563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 772.181881] LNet: Removed LNI 192.168.203.33@tcp [ 772.819594] Key type .llcrypt unregistered [ 772.821388] Key type ._llcrypt unregistered [ 787.564920] Key type ._llcrypt registered [ 787.570214] Key type .llcrypt registered [ 789.069417] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 09:43:03 (1789566183) [ 790.043515] Key type .llcrypt unregistered [ 790.049860] Key type ._llcrypt unregistered [ 797.405293] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 797.643786] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 797.659565] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 797.672850] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 800.360951] Key type ._llcrypt registered [ 800.363124] Key type .llcrypt registered [ 800.577056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 800.612724] alg: No test for adler32 (adler32-zlib) [ 801.576923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.632841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 801.651154] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 802.642598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 802.673448] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 806.473205] LNet: 38053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.477772] LNetError: 38053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 806.487760] LNet: Removed LNI 10.1.2.3@tcp [ 806.499721] LNet: Removed LNI 192.168.203.33@tcp [ 807.139793] Key type .llcrypt unregistered [ 807.144069] Key type ._llcrypt unregistered [ 821.849443] Key type ._llcrypt registered [ 821.853966] Key type .llcrypt registered [ 823.613812] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 09:43:37 (1789566217) [ 824.229447] Key type .llcrypt unregistered [ 824.231169] Key type ._llcrypt unregistered [ 831.206926] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 831.501709] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 831.504972] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 831.508699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 834.323395] Key type ._llcrypt registered [ 834.333350] Key type .llcrypt registered [ 834.606136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 834.634968] alg: No test for adler32 (adler32-zlib) [ 835.637442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.730427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 835.761856] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 838.678259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 838.744830] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 842.902657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 842.929653] LNet: Removed LNI 10.1.2.3@tcp [ 842.962121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 842.992267] LNet: Removed LNI 10.1.2.31@tcp [ 843.013358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 843.064805] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 844.307739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 844.346880] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 845.752719] LNet: 39986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.766534] LNetError: 39986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 845.776412] LNet: Removed LNI 10.1.2.31@tcp [ 845.789868] LNet: Removed LNI 10.1.2.3@tcp [ 846.650715] Key type .llcrypt unregistered [ 846.652431] Key type ._llcrypt unregistered [ 862.597140] Key type ._llcrypt registered [ 862.600772] Key type .llcrypt registered [ 864.375790] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:44:18 (1789566258) [ 865.033563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 865.059073] alg: No test for adler32 (adler32-zlib) [ 866.230748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.400648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 866.441697] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 867.556068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 867.588882] LNet: 40937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.598958] LNetError: 40937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.616271] LNet: Removed LNI 192.168.203.33@tcp [ 867.661689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.716574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.33@tcp [ 867.753881] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 868.884272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.33@tcp [ 868.934726] LNet: Removed LNI 192.168.203.33@tcp [ 869.026759] LNet: 41072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.871712] Key type .llcrypt unregistered [ 869.873347] Key type ._llcrypt unregistered [ 886.224182] Key type ._llcrypt registered [ 886.232757] Key type .llcrypt registered [ 888.225520] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:44:42 (1789566282) [ 890.823530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg333-server [ 893.324957] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:44:47 (1789566287) [ 893.963382] Key type .llcrypt unregistered [ 893.964979] Key type ._llcrypt unregistered [ 901.008938] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 901.357497] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 901.362626] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 901.366983] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 904.144474] Key type ._llcrypt registered [ 904.147430] Key type .llcrypt registered [ 904.559863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 904.599412] alg: No test for adler32 (adler32-zlib) [ 905.577990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.679485] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 906.936734] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 910.091349] LNet: 42906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.102886] LNetError: 42906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.110334] LNet: Removed LNI 192.168.203.33@tcp1 [ 910.124949] LNet: Removed LNI 10.1.2.3@tcp2 [ 910.989506] Key type .llcrypt unregistered [ 910.995298] Key type ._llcrypt unregistered [ 924.943244] Key type ._llcrypt registered [ 924.945815] Key type .llcrypt registered [ 926.158534] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:45:21 (1789566321) [ 926.571370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 926.580605] alg: No test for adler32 (adler32-zlib) [ 927.526411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.565457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 927.583464] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 928.401271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 928.473804] LNet: 43873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.480770] LNetError: 43873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.520585] LNet: Removed LNI 192.168.203.33@tcp [ 928.564834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.679686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 928.713948] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 937.047600] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:45:31 (1789566331) [ 937.181599] LNet: 44455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.206210] LNetError: 44455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.228422] LNet: Removed LNI 192.168.203.33@tcp [ 937.936630] Key type .llcrypt unregistered [ 937.942634] Key type ._llcrypt unregistered [ 944.904958] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 945.280698] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 945.288076] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 945.295701] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 948.094523] Key type ._llcrypt registered [ 948.095995] Key type .llcrypt registered [ 948.375466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 948.389718] alg: No test for adler32 (adler32-zlib) [ 949.432698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.516788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 949.548291] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 950.485942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 950.521862] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 952.722682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.746958] LNet: 45203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.762676] LNetError: 45203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 952.775813] LNet: Removed LNI 10.1.2.3@tcp [ 952.787716] LNet: Removed LNI 192.168.203.33@tcp [ 952.826590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.916683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 952.967505] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 952.975343] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 953.671645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.724580] LNet: 45230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.731541] LNetError: 45230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 953.743417] LNetError: 45230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 953.759632] LNet: Removed LNI 10.1.2.3@tcp [ 953.787278] LNet: Removed LNI 192.168.203.33@tcp [ 953.829689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.944135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 953.999912] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 954.012484] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 954.743365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 954.777978] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 956.074582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.093212] LNet: 45393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.099381] LNetError: 45393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 956.105285] LNetError: 45393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 956.119034] LNet: Removed LNI 10.1.2.3@tcp [ 956.127908] LNet: Removed LNI 192.168.203.33@tcp [ 956.143385] LNet: Removed LNI 192.168.203.33@tcp2 [ 956.193443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.300759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 956.349359] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 956.372572] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 956.379767] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 965.373746] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 09:45:59 (1789566359) [ 965.510817] LNet: 45982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.525920] LNetError: 45982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 965.535990] LNetError: 45982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 965.547724] LNet: Removed LNI 10.1.2.3@tcp [ 965.552888] LNet: Removed LNI 192.168.203.33@tcp [ 965.572111] LNet: Removed LNI 192.168.203.33@tcp2 [ 966.306575] Key type .llcrypt unregistered [ 966.312648] Key type ._llcrypt unregistered [ 974.479431] Key type ._llcrypt registered [ 974.486106] Key type .llcrypt registered [ 974.700941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 974.725363] alg: No test for adler32 (adler32-zlib) [ 975.798152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.873695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 975.963187] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 976.933427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 976.967168] LNet: 46382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.989969] LNetError: 46382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.010173] LNet: Removed LNI 192.168.203.33@tcp [ 977.067905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.184789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 977.226891] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 977.991662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.032076] LNet: 46412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.060775] LNetError: 46412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.116684] LNet: Removed LNI 192.168.203.33@tcp [ 978.203632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.318723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 978.366172] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 979.397754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 979.421215] LNet: 46442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.436635] LNetError: 46442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.449749] LNet: Removed LNI 192.168.203.33@tcp [ 979.483177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.559498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 979.593612] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 980.635356] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 980.836799] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 980.870147] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 980.875159] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 983.407329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.436850] LNet: 46520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.448695] LNetError: 46520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.469836] LNet: Removed LNI 192.168.203.33@tcp [ 983.507819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.575828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 983.606621] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 993.405469] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 09:46:27 (1789566387) [ 993.895102] LNet: 47117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.900934] LNetError: 47117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.977316] LNet: Removed LNI 192.168.203.33@tcp [ 995.217840] Key type .llcrypt unregistered [ 995.222161] Key type ._llcrypt unregistered [ 1005.953604] Key type ._llcrypt registered [ 1005.956640] Key type .llcrypt registered [ 1006.208534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1006.243582] alg: No test for adler32 (adler32-zlib) [ 1007.226189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.322908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1007.376570] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1008.749814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1008.794838] LNet: 47513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.805630] LNetError: 47513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.832818] LNet: Removed LNI 192.168.203.33@tcp [ 1008.889212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.038598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1009.069950] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1018.878713] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 09:46:53 (1789566413) [ 1019.085850] LNet: 48100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.093715] LNetError: 48100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.159166] LNet: Removed LNI 192.168.203.33@tcp [ 1020.198217] Key type .llcrypt unregistered [ 1020.200739] Key type ._llcrypt unregistered [ 1028.902028] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1029.296616] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1029.311268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1029.318795] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1032.225352] Key type ._llcrypt registered [ 1032.231380] Key type .llcrypt registered [ 1032.564704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1032.617129] alg: No test for adler32 (adler32-zlib) [ 1033.712447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1033.867367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1033.923146] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1035.418286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1035.441832] LNet: 48551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.446447] LNetError: 48551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.471689] LNet: Removed LNI 192.168.203.33@tcp [ 1035.506031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1035.620486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1035.694531] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1045.532073] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 09:47:19 (1789566439) [ 1045.655626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1045.688925] LNet: 49147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.693393] LNetError: 49147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1045.717417] LNet: Removed LNI 192.168.203.33@tcp [ 1045.751441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1045.995762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1054.527352] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 09:47:28 (1789566448) [ 1054.967965] LNet: 49736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.754183] Key type .llcrypt unregistered [ 1055.760824] Key type ._llcrypt unregistered [ 1064.636477] Key type ._llcrypt registered [ 1064.640605] Key type .llcrypt registered [ 1064.870618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1064.883702] alg: No test for adler32 (adler32-zlib) [ 1065.851750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1065.962569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1066.026238] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1066.897484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1066.921823] LNet: 50130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.926675] LNetError: 50130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.935760] LNet: Removed LNI 192.168.203.33@tcp [ 1066.954839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1067.014569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1067.032551] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1077.337735] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 09:47:51 (1789566471) [ 1077.547067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1077.598869] LNet: 50719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.606463] LNetError: 50719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.636825] LNet: Removed LNI 192.168.203.33@tcp [ 1077.689893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1077.818172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1085.681223] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 09:48:00 (1789566480) [ 1087.894519] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1089.994769] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 09:48:04 (1789566484) [ 1092.804708] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1095.339530] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 09:48:09 (1789566489) [ 1097.765462] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1101.071912] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 09:48:14 (1789566494) [ 1103.554191] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1106.366566] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 09:48:20 (1789566500) [ 1109.342675] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1111.909513] Lustre: DEBUG MARKER: == sanity-lnet test 163: Check import of net with mixed IPv6/IPv4 explicit NIDs ========================================================== 09:48:26 (1789566506) [ 1112.043605] LNet: 52740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.679785] Key type .llcrypt unregistered [ 1112.685181] Key type ._llcrypt unregistered [ 1122.595787] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1122.902022] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1122.932695] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1122.954091] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1126.046725] Key type ._llcrypt registered [ 1126.051383] Key type .llcrypt registered [ 1126.435270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1126.456378] alg: No test for adler32 (adler32-zlib) [ 1127.456702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1127.590386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-163-expected.yaml [ 1127.654105] LNet: Added LNI 2001:db8:0:f101::1@tcp [8/256/0/180] [ 1127.666857] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1127.845062] LNet: 53088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.856663] LNetError: 53088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1127.874467] LNet: Removed LNI 2001:db8:0:f101::1@tcp [ 1127.895357] LNet: Removed LNI 10.1.2.3@tcp1 [ 1128.613620] Key type .llcrypt unregistered [ 1128.629353] Key type ._llcrypt unregistered [ 1144.892360] Key type ._llcrypt registered [ 1144.897848] Key type .llcrypt registered [ 1146.714795] Lustre: DEBUG MARKER: == sanity-lnet test 164: Check no issue when reading 'nid' in all scenarios ========================================================== 09:49:00 (1789566540) [ 1147.300427] Key type .llcrypt unregistered [ 1147.305072] Key type ._llcrypt unregistered [ 1155.743099] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1155.976776] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1158.653559] Key type ._llcrypt registered [ 1158.656759] Key type .llcrypt registered [ 1158.923201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1158.950717] alg: No test for adler32 (adler32-zlib) [ 1159.936211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1161.887629] Lustre: DEBUG MARKER: Check malformed nid fails [ 1162.212894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1162.382056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1162.425704] LNet: 54326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1162.448217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1165.037531] Lustre: DEBUG MARKER: Check wrong net in nid fails [ 1165.163573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1165.207834] LNetError: 54464:0:(api-ni.c:4164:lnet_dyn_add_ni()) NID network tcp2 does not match configured network tcp1 [ 1165.342631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1165.394048] LNet: 54474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.460290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1165.575297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1165.621848] LNetError: 54483:0:(config.c:1721:lnet_inet_select()) ksocklnd: failed to find UP interface with IP address 192.168.203.33@tcp1 [ 1165.658611] LNetError: Error -100 starting up LNI tcp [ 1165.752806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1165.823576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1168.070986] Lustre: DEBUG MARKER: Check 'nid' cannot be used in ip2nets [ 1168.193124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1168.378386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1168.423792] LNet: 54640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.444282] LNet: 54640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1168.501992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1171.196826] Lustre: DEBUG MARKER: Check MR works [ 1171.308087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1171.388261] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1171.407753] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1171.557311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1171.583831] LNet: 54802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.599856] LNetError: 54802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1171.622227] LNet: Removed LNI 10.1.2.3@tcp [ 1171.672272] LNet: Removed LNI 192.168.203.33@tcp [ 1171.727883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1174.130568] Lustre: DEBUG MARKER: Check nid/interfaces order doesn't matter [ 1174.334313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1174.464578] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1174.517412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp [ 1174.638462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1174.676678] LNetError: 54966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1174.690402] LNetError: 54966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1174.721455] LNet: Removed LNI 192.168.203.33@tcp [ 1174.755909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1175.938429] Key type .llcrypt unregistered [ 1175.940182] Key type ._llcrypt unregistered [ 1191.068234] Key type ._llcrypt registered [ 1191.075308] Key type .llcrypt registered [ 1192.835931] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing net ========================================================== 09:49:47 (1789566587) [ 1193.539315] Key type .llcrypt unregistered [ 1193.543954] Key type ._llcrypt unregistered [ 1202.352062] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1202.680651] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1205.579846] Key type ._llcrypt registered [ 1205.586403] Key type .llcrypt registered [ 1205.986688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1206.005725] alg: No test for adler32 (adler32-zlib) [ 1207.021948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1207.141930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1207.251666] LNet: Added LNI 192.168.203.33@tcp [8/512/0/180] [ 1207.278657] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1207.434521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1207.469715] LNet: 56095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1207.496111] LNetError: 56095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1207.509460] LNet: Removed LNI 10.1.2.3@tcp [ 1207.519666] LNetError: 56095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1207.544963] LNet: Removed LNI 192.168.203.33@tcp [ 1207.629583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1207.749611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1207.819558] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1207.965329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1208.026783] LNet: 56119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1208.044416] LNetError: 56119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1208.083232] LNet: Removed LNI 192.168.203.33@tcp [ 1208.117870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1208.229307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1208.270555] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1208.280831] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1208.427359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1208.482734] LNet: 56145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1208.494900] LNetError: 56145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1208.509745] LNet: Removed LNI 10.1.2.3@tcp [ 1208.521971] LNetError: 56145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1208.553722] LNet: Removed LNI 192.168.203.33@tcp [ 1208.644475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1208.795655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1208.827915] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1208.831727] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1208.833651] LNet: 56153:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1209.274110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1209.331439] LNet: 56179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.343313] LNetError: 56179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1209.365413] LNet: Removed LNI 10.1.2.3@tcp [ 1209.368390] LNetError: 56179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.384833] LNet: Removed LNI 192.168.203.33@tcp [ 1209.432390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1209.570617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1209.639270] LNet: Added LNI 192.168.203.33@tcp [8/256/0/182] [ 1209.660881] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1209.890447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1209.943393] LNet: 56207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.954124] LNetError: 56207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1209.968815] LNet: Removed LNI 10.1.2.3@tcp [ 1209.970422] LNetError: 56207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1210.039684] LNet: Removed LNI 192.168.203.33@tcp [ 1210.121183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1210.321969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1210.395340] LNet: Added LNI 192.168.203.33@tcp [8/256/0/182] [ 1210.404241] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1210.406152] LNet: 56215:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1210.702521] LNet: Removed LNI 10.1.2.3@tcp [ 1210.715115] LNet: Removed LNI 192.168.203.33@tcp [ 1211.828296] Key type .llcrypt unregistered [ 1211.847232] Key type ._llcrypt unregistered [ 1230.483968] Key type ._llcrypt registered [ 1230.485950] Key type .llcrypt registered [ 1232.364340] Lustre: DEBUG MARKER: == sanity-lnet test 171: Check CPTs and tunables parsing when importing net (old-api) ========================================================== 09:50:26 (1789566626) [ 1233.342353] Key type .llcrypt unregistered [ 1233.343934] Key type ._llcrypt unregistered [ 1242.508821] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1242.837911] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1245.612637] Key type ._llcrypt registered [ 1245.617618] Key type .llcrypt registered [ 1245.952077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1245.972108] alg: No test for adler32 (adler32-zlib) [ 1246.877287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1246.976569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1247.005845] LNet: Added LNI 192.168.203.33@tcp [8/512/0/180] [ 1247.018511] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1247.021151] LNet: 57326:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1247.125554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1247.150407] LNet: 57345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.157763] LNetError: 57345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1247.164029] LNet: Removed LNI 10.1.2.3@tcp [ 1247.166713] LNetError: 57345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1247.201295] LNet: Removed LNI 192.168.203.33@tcp [ 1247.238437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1247.285300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1247.336631] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1247.458713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1247.514877] LNet: 57369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.549068] LNetError: 57369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1247.595405] LNet: Removed LNI 192.168.203.33@tcp [ 1247.660670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1247.781626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1247.846598] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1247.861907] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1247.872891] LNet: 57377:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1248.107873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1248.183673] LNet: 57395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.212076] LNetError: 57395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1248.232271] LNet: Removed LNI 10.1.2.3@tcp [ 1248.237591] LNetError: 57395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1248.270812] LNet: Removed LNI 192.168.203.33@tcp [ 1248.334279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1248.596740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1248.695800] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1248.719525] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1248.721506] LNet: 57403:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1249.236895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1249.294344] LNet: 57429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1249.327326] LNetError: 57429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1249.336857] LNet: Removed LNI 10.1.2.3@tcp [ 1249.341888] LNetError: 57429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.399139] LNet: Removed LNI 192.168.203.33@tcp [ 1249.461989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1249.726924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1249.844962] LNet: Added LNI 192.168.203.33@tcp [8/256/0/182] [ 1249.867436] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1250.128061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1250.197086] LNet: 57457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1250.219467] LNetError: 57457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1250.236783] LNet: Removed LNI 10.1.2.3@tcp [ 1250.238451] LNetError: 57457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1250.283664] LNet: Removed LNI 192.168.203.33@tcp [ 1250.345229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1250.431989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1250.451046] LNet: Added LNI 192.168.203.33@tcp [8/256/0/182] [ 1250.454730] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1250.460931] LNet: 57465:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1250.580661] LNet: Removed LNI 10.1.2.3@tcp [ 1250.591956] LNet: Removed LNI 192.168.203.33@tcp [ 1251.255749] Key type .llcrypt unregistered [ 1251.260292] Key type ._llcrypt unregistered [ 1271.482128] Key type ._llcrypt registered [ 1271.483663] Key type .llcrypt registered [ 1273.337681] Lustre: DEBUG MARKER: == sanity-lnet test 172: Check CPTs and tunables parsing when importing ip2nets ========================================================== 09:51:07 (1789566667) [ 1274.026437] Key type .llcrypt unregistered [ 1274.032592] Key type ._llcrypt unregistered [ 1282.332283] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1282.600873] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1285.459396] Key type ._llcrypt registered [ 1285.464070] Key type .llcrypt registered [ 1285.806982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1285.825625] alg: No test for adler32 (adler32-zlib) [ 1286.885158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1287.051925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1287.114405] LNet: Added LNI 192.168.203.33@tcp [8/432/0/182] [ 1287.130611] LNet: Added LNI 10.1.2.3@tcp1 [8/402/0/181] [ 1287.588390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1287.624354] LNet: 58607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1287.639480] LNetError: 58607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1287.652603] LNet: Removed LNI 192.168.203.33@tcp [ 1287.667443] LNet: Removed LNI 10.1.2.3@tcp1 [ 1287.734689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1287.886794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1287.944755] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1287.954880] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/181] [ 1288.233388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1288.301579] LNet: 58639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.318791] LNetError: 58639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.329830] LNetError: 58639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1288.344194] LNet: Removed LNI 192.168.203.33@tcp [ 1288.393559] LNet: Removed LNI 10.1.2.3@tcp1 [ 1288.460254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1288.637595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1288.700487] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1288.716202] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1288.872456] LNet: Removed LNI 10.1.2.3@tcp [ 1288.898919] LNet: Removed LNI 192.168.203.33@tcp [ 1290.164542] Key type .llcrypt unregistered [ 1290.171315] Key type ._llcrypt unregistered [ 1307.814899] Key type ._llcrypt registered [ 1307.820723] Key type .llcrypt registered [ 1309.830070] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 09:51:43 (1789566703) [ 1312.110639] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1314.661529] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:51:48 (1789566708) [ 1315.274404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1315.294808] alg: No test for adler32 (adler32-zlib) [ 1316.388719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1316.500167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1316.548418] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1317.884480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.33 [ 1317.911501] LNet: 59908:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1317.939688] LNet: Removed LNI 192.168.203.33@tcp [ 1327.174604] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:52:01 (1789566721) [ 1327.333894] LNet: 60470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1327.368147] LNetError: 60470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1327.417033] LNet: Removed LNI 192.168.203.33@tcp [ 1328.509412] Key type .llcrypt unregistered [ 1328.512228] Key type ._llcrypt unregistered [ 1337.733901] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1338.180037] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1341.248778] Key type ._llcrypt registered [ 1341.253906] Key type .llcrypt registered [ 1341.687378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1341.737349] alg: No test for adler32 (adler32-zlib) [ 1343.467496] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1352.162920] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:52:26 (1789566746) [ 1352.436209] LNet: 61388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1352.456106] LNetError: 61388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1352.508377] LNet: Removed LNI 10.1.2.3@tcp [ 1353.413987] Key type .llcrypt unregistered [ 1353.417146] Key type ._llcrypt unregistered [ 1361.664160] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1362.047080] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1362.048865] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1362.061788] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1365.086920] Key type ._llcrypt registered [ 1365.090770] Key type .llcrypt registered [ 1365.433365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1365.461582] alg: No test for adler32 (adler32-zlib) [ 1366.555147] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1375.624372] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:52:50 (1789566770) [ 1378.272227] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1380.697900] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:52:55 (1789566775) [ 1380.821153] LNet: 62567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1380.830829] LNetError: 62567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1380.870289] LNet: Removed LNI 10.1.2.3@tcp [ 1381.775155] Key type .llcrypt unregistered [ 1381.778984] Key type ._llcrypt unregistered [ 1389.949040] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1390.306256] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1390.310767] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1390.318982] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1393.231561] Key type ._llcrypt registered [ 1393.236696] Key type .llcrypt registered [ 1393.592217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1393.623588] alg: No test for adler32 (adler32-zlib) [ 1394.686927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1394.781462] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1406.467201] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:53:20 (1789566800) [ 1407.059111] LNet: 63803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1407.065695] LNetError: 63803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1407.090507] LNet: Removed LNI 10.1.2.3@tcp [ 1408.095609] Key type .llcrypt unregistered [ 1408.101980] Key type ._llcrypt unregistered [ 1416.696343] Key type ._llcrypt registered [ 1416.704293] Key type .llcrypt registered [ 1417.001629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1417.038639] alg: No test for adler32 (adler32-zlib) [ 1418.106440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1418.156382] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1426.918048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1427.837995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_interrupt -r 1 [ 1427.884747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_interrupt -r 1 [ 1427.931949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1427.963077] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1428.327857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1428.394740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1428.461471] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 1428.465393] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1428.471479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1428.515461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1428.577044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1428.624847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1428.670763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1428.714344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1428.765143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1428.797678] LNet: 64241:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1428.831510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1428.887193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1428.961650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1429.023598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1429.101956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1429.157626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1429.212143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1430.111506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_dropped -r 1 [ 1430.168468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_dropped -r 1 [ 1430.219846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1430.248777] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1430.552531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1430.578641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1430.605981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1430.631519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1430.672391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1430.722324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1430.768047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1430.838606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1430.902856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1430.943392] LNet: 64352:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1430.974502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1431.034738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1431.095149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1431.136876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1431.181308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1431.218597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1431.253868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1432.015794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_aborted -r 1 [ 1432.083273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_aborted -r 1 [ 1432.129204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1432.387706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1432.419847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1432.461771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1432.500862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1432.542453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1432.576487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1432.625920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1432.688939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1432.737360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1432.761687] LNet: 64463:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1432.798327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1432.835074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1432.869770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1432.925128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1433.002402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1433.102389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1433.190425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1433.936612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_no_route -r 1 [ 1433.976259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_no_route -r 1 [ 1434.024602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1434.040927] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1434.044324] LNet: Skipped 1 previous similar message [ 1434.330403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1434.382368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1434.448857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1434.497887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1434.562779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1434.631847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1434.694186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1434.747738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1434.790569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1434.811820] LNet: 64575:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1434.841488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1434.902807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1434.940466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1434.983891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1435.026539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1435.061722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1435.115768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1435.820596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_timeout -r 1 [ 1435.895507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_timeout -r 1 [ 1435.972341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1436.378383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1436.441754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1436.487529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1436.560296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1436.649582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1436.719655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1436.786802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1436.831031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1436.894668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1436.947249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1437.006859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1437.046900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1437.107307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1437.143907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1437.187499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1437.222694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1437.881479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_error -r 1 [ 1437.927903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_error -r 1 [ 1438.031309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1438.085056] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1438.092062] LNet: Skipped 1 previous similar message [ 1438.461802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1438.520141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1438.584842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1438.637462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1438.686275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1438.720832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1438.778688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1438.817824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1438.867941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1438.901969] LNet: 64797:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1438.908783] LNet: 64797:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1438.948421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1438.989681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1439.029862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1439.115115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1439.213946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1439.290475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1439.346464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1439.722542] LNet: 64838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1439.740544] LNetError: 64838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.765752] LNet: Removed LNI 192.168.203.33@tcp [ 1440.498861] Key type .llcrypt unregistered [ 1440.505564] Key type ._llcrypt unregistered [ 1463.553375] Key type ._llcrypt registered [ 1463.569204] Key type .llcrypt registered [ 1465.533602] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:54:19 (1789566859) [ 1466.965369] Key type .llcrypt unregistered [ 1466.971561] Key type ._llcrypt unregistered [ 1475.076478] Key type ._llcrypt registered [ 1475.078156] Key type .llcrypt registered [ 1475.380801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1475.416280] alg: No test for adler32 (adler32-zlib) [ 1476.513877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1476.548412] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1486.494701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1499.514482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.558860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1499.598943] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1501.671705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_interrupt -r 1 [ 1501.703988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_interrupt -r 1 [ 1501.750456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_interrupt -r 1 [ 1501.817440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_interrupt -r 1 [ 1501.855339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_interrupt -r 1 [ 1501.881900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_interrupt -r 1 [ 1501.912966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_interrupt -r 1 [ 1501.941948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_interrupt -r 1 [ 1501.971970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1501.989148] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1501.994723] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 1502.190945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1502.218191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1502.250513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1502.284279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1502.315896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1502.340499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1502.386465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1502.439494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1502.490133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1502.512425] LNet: 66305:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1502.561824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1502.588871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1502.631432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1502.702958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1502.743442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1502.786114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1502.844335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1503.746662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_dropped -r 1 [ 1503.802851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_dropped -r 1 [ 1503.857839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_dropped -r 1 [ 1503.906466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_dropped -r 1 [ 1503.965296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_dropped -r 1 [ 1503.999823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_dropped -r 1 [ 1504.033691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_dropped -r 1 [ 1504.076873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_dropped -r 1 [ 1504.113478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1504.145477] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1504.150542] LNet: Skipped 2 previous similar messages [ 1504.455361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1504.520171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1504.558883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1504.594304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1504.622923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1504.653583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1504.685977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1504.721828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1504.775750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1504.806596] LNet: 66428:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1504.838237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1504.868391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1504.898565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1504.938236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1504.979431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1505.039233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1505.088844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1505.893833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_aborted -r 1 [ 1505.941515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_aborted -r 1 [ 1506.014493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_aborted -r 1 [ 1506.047257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_aborted -r 1 [ 1506.075798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_aborted -r 1 [ 1506.135487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_aborted -r 1 [ 1506.179661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_aborted -r 1 [ 1506.250932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_aborted -r 1 [ 1506.292951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1506.310857] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1506.321051] LNet: Skipped 2 previous similar messages [ 1506.550465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1506.589873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1506.628928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1506.667543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1506.698855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1506.736805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1506.776402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1506.827142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1506.869583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1506.902030] LNet: 66552:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1506.920248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1506.981692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1507.071525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1507.106778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1507.160206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1507.202566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1507.282663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1507.999639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_no_route -r 1 [ 1508.055549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_no_route -r 1 [ 1508.111723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_no_route -r 1 [ 1508.155876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_no_route -r 1 [ 1508.202247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_no_route -r 1 [ 1508.256823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_no_route -r 1 [ 1508.335436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_no_route -r 1 [ 1508.385395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_no_route -r 1 [ 1508.455779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1508.483309] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1508.487922] LNet: Skipped 2 previous similar messages [ 1508.821912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1508.871153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1508.919459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1508.975733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1509.026826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1509.060936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1509.120510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1509.199788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1509.266143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1509.286155] LNet: 66675:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1509.309137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1509.372171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1509.415830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1509.459876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1509.519794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1509.591792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1509.651517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1510.454787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_timeout -r 1 [ 1510.502364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_timeout -r 1 [ 1510.545954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_timeout -r 1 [ 1510.615879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_timeout -r 1 [ 1510.664553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_timeout -r 1 [ 1510.705234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_timeout -r 1 [ 1510.771361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_timeout -r 1 [ 1510.829900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_timeout -r 1 [ 1510.907720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1511.363537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1511.419746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1511.505934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1511.573228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1511.620652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1511.680330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1511.745907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1511.812538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1511.850857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1511.878953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1511.918101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1511.957521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1512.027202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1512.081839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1512.147403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1512.229418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1513.094056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_error -r 1 [ 1513.158663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_error -r 1 [ 1513.244394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_error -r 1 [ 1513.326198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_error -r 1 [ 1513.415417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_error -r 1 [ 1513.488272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_error -r 1 [ 1513.593300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_error -r 1 [ 1513.733926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_error -r 1 [ 1513.844342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1513.890229] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1513.908118] LNet: Skipped 7 previous similar messages [ 1514.348460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1514.419688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1514.464637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1514.532085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1514.603157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1514.666295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1514.739695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1514.861409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1514.954340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1514.977349] LNet: 66922:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1514.992265] LNet: 66922:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1515.036785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1515.068654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1515.095956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1515.150207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1515.197982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1515.235363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1515.259773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1521.831637] LNet: 66998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.838454] LNetError: 66998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1521.847923] LNet: Removed LNI 192.168.203.33@tcp [ 1522.931218] LNet: Removed LNI 192.168.203.33@tcp1 [ 1523.618791] Key type .llcrypt unregistered [ 1523.626488] Key type ._llcrypt unregistered [ 1546.641521] Key type ._llcrypt registered [ 1546.644308] Key type .llcrypt registered [ 1548.649820] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:55:42 (1789566942) [ 1549.666688] Key type .llcrypt unregistered [ 1549.668792] Key type ._llcrypt unregistered [ 1558.415199] Key type ._llcrypt registered [ 1558.420446] Key type .llcrypt registered [ 1558.838141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1558.858241] alg: No test for adler32 (adler32-zlib) [ 1559.922453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1559.953087] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1569.621311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1570.625607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e remote_dropped -r 1 [ 1570.705266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e remote_dropped -r 1 [ 1570.773468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1570.809537] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1571.150833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1571.214232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1571.249744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1571.312648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1571.377480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1571.410395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1571.467450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1571.510743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1571.561441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1571.593678] LNet: 68245:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1571.618039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1571.672957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1571.716967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1571.795154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1571.867616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1571.934733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1571.987653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1572.826347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e remote_error -r 1 [ 1572.881709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e remote_error -r 1 [ 1572.912820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1572.930855] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1573.317821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1573.349529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1573.401228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1573.457590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1573.519555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1573.585395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1573.638948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1573.672433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1573.732381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1573.769563] LNet: 68356:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1573.797979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1573.863269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1573.927420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1573.998681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1574.063406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1574.133685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1574.228165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1575.210191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e remote_timeout -r 1 [ 1575.275256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e remote_timeout -r 1 [ 1575.323890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1575.381503] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1575.723576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1575.756629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1575.815604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1575.852532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1575.881804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1575.911855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1575.945835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1575.996801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1576.039507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1576.058396] LNet: 68468:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1576.074289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1576.106410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1576.153397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1576.197713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1576.234972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1576.287902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1576.346568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1576.723868] LNet: 68509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.734954] LNetError: 68509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1576.775760] LNet: Removed LNI 192.168.203.33@tcp [ 1577.470101] Key type .llcrypt unregistered [ 1577.471886] Key type ._llcrypt unregistered [ 1601.743897] Key type ._llcrypt registered [ 1601.747229] Key type .llcrypt registered [ 1603.868906] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:56:37 (1789566997) [ 1605.129388] Key type .llcrypt unregistered [ 1605.133249] Key type ._llcrypt unregistered [ 1614.516389] Key type ._llcrypt registered [ 1614.519271] Key type .llcrypt registered [ 1614.978230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1615.011718] alg: No test for adler32 (adler32-zlib) [ 1615.988618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1616.020537] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1624.168562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1634.881748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.930268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1635.009343] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1637.190715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e remote_dropped -r 1 [ 1637.230613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e remote_dropped -r 1 [ 1637.284284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e remote_dropped -r 1 [ 1637.318440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e remote_dropped -r 1 [ 1637.350760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e remote_dropped -r 1 [ 1637.387417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e remote_dropped -r 1 [ 1637.428957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e remote_dropped -r 1 [ 1637.509167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e remote_dropped -r 1 [ 1637.598965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1637.634390] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1637.649483] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.133@tcp [ 1638.016733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1638.044701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1638.069185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1638.127228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1638.154363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1638.190262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1638.255699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1638.326940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1638.399147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1638.425779] LNet: 69975:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1638.451198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1638.504720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1638.587353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1638.617719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1638.689215] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1638.707368] LNet: Skipped 3 previous similar messages [ 1638.719611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1638.775891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1638.827866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1638.875048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1639.611654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e remote_error -r 1 [ 1639.688129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e remote_error -r 1 [ 1639.802914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e remote_error -r 1 [ 1639.907170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e remote_error -r 1 [ 1640.003041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e remote_error -r 1 [ 1640.084966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e remote_error -r 1 [ 1640.176821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e remote_error -r 1 [ 1640.247186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e remote_error -r 1 [ 1640.325557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1640.372825] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1640.730559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1640.783497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1640.886156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1640.934420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1641.005180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1641.101716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1641.167393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1641.223704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1641.303317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1641.349259] LNet: 70101:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1641.385155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1641.453262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1641.509451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1641.591339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1641.675107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1641.733974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1641.788764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1641.830558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1642.646994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e remote_timeout -r 1 [ 1642.717947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e remote_timeout -r 1 [ 1642.776045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e remote_timeout -r 1 [ 1642.830339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e remote_timeout -r 1 [ 1642.905233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e remote_timeout -r 1 [ 1642.994873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e remote_timeout -r 1 [ 1643.067738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e remote_timeout -r 1 [ 1643.160239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e remote_timeout -r 1 [ 1643.211897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1643.255714] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1643.578911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1643.660301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1643.720656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1643.824370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1643.933299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1644.001917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1644.069358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1644.113332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1644.166559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1644.191277] LNet: 70226:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1644.230446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1644.286989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1644.343253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1644.401222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1644.458129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1644.516918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1644.585470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1644.644924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1652.060148] LNet: 70304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.086327] LNetError: 70304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1652.099994] LNet: Removed LNI 192.168.203.33@tcp [ 1652.135989] LNet: Removed LNI 192.168.203.33@tcp1 [ 1652.729043] Key type .llcrypt unregistered [ 1652.730992] Key type ._llcrypt unregistered [ 1673.157424] Key type ._llcrypt registered [ 1673.163334] Key type .llcrypt registered [ 1675.030386] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:57:49 (1789567069) [ 1677.486301] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1679.656957] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:57:54 (1789567074) [ 1681.043575] Key type .llcrypt unregistered [ 1681.047888] Key type ._llcrypt unregistered [ 1689.756815] Key type ._llcrypt registered [ 1689.758210] Key type .llcrypt registered [ 1690.054069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1690.071491] alg: No test for adler32 (adler32-zlib) [ 1691.005506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1691.029093] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1700.491720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1701.237237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e network_timeout -r 1 [ 1701.364520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e network_timeout -r 1 [ 1701.451633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1701.479795] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1702.031506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1702.077478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1702.124336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1702.178363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1702.238681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1702.293604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1702.304396] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 1702.322467] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1702.397500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1702.470818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1702.539722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1702.563292] LNet: 71847:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1702.586483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1702.635580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1702.666892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1702.726489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1702.780840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1702.852438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1703.259965] LNet: 71886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.268504] LNetError: 71886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.294797] LNet: Removed LNI 192.168.203.33@tcp [ 1703.809685] Key type .llcrypt unregistered [ 1703.815535] Key type ._llcrypt unregistered [ 1729.072844] Key type ._llcrypt registered [ 1729.076460] Key type .llcrypt registered [ 1729.495052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1729.523519] alg: No test for adler32 (adler32-zlib) [ 1730.578708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1730.637641] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1739.652255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1753.187945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.287706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1753.330555] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1755.761670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e network_timeout -r 1 [ 1755.833755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e network_timeout -r 1 [ 1755.898904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e network_timeout -r 1 [ 1755.982591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e network_timeout -r 1 [ 1756.067503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e network_timeout -r 1 [ 1756.161526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e network_timeout -r 1 [ 1756.246078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e network_timeout -r 1 [ 1756.349225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e network_timeout -r 1 [ 1756.435808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1756.484618] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1756.949197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1757.017702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1757.085793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1757.158276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1757.250568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1757.289968] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1757.290284] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 1757.304319] LNet: Skipped 1 previous similar message [ 1757.329610] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.133@tcp [ 1757.331974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1757.389636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1757.454170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1757.510729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1757.540406] LNet: 72764:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1757.579867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1757.651338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1757.698599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1757.757540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1757.842843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1757.913266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1763.391652] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1763.842073] LNet: 72838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.852276] LNetError: 72838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.865954] LNet: Removed LNI 192.168.203.33@tcp [ 1763.900406] LNet: Removed LNI 192.168.203.33@tcp1 [ 1764.473773] Key type .llcrypt unregistered [ 1764.480241] Key type ._llcrypt unregistered [ 1786.564781] Key type ._llcrypt registered [ 1786.569862] Key type .llcrypt registered [ 1788.625246] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:59:42 (1789567182) [ 1789.225134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1789.262209] alg: No test for adler32 (adler32-zlib) [ 1790.321277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1790.453297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1790.493251] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1791.729461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1791.762383] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1792.867814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 1793.006190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1793.085949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1793.234807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1793.311276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1793.349785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1793.380247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.33@tcp [ 1793.395570] LNetError: 73938:0:(api-ni.c:4493:lnet_ni_set_healthv()) manually adding local NI 192.168.203.33@tcp to recovery [ 1793.440707] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 1793.447418] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1795.487357] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1796.012378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1799.587586] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1800.954506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1800.998723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1801.117782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.173182] LNet: 74045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.179556] LNetError: 74045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.190069] LNet: Removed LNI 192.168.203.33@tcp [ 1801.217510] LNet: Removed LNI 192.168.203.33@tcp1 [ 1801.270177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.369829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1801.423685] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1802.503518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1802.550597] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1803.806762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 1803.906327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1804.021817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1804.097440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1804.139855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1804.236203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1804.289785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1804.364352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.33@tcp [ 1804.399208] LNetError: 74317:0:(api-ni.c:4493:lnet_ni_set_healthv()) manually adding local NI 192.168.203.33@tcp to recovery [ 1804.449679] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1807.108576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1810.593045] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1810.607686] LNet: Skipped 1 previous similar message [ 1815.662829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1815.752924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1815.827533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1825.040533] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:00:19 (1789567219) [ 1825.154309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1825.228084] LNet: 75021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.243128] LNetError: 75021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.251430] LNetError: 75021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1825.258694] LNet: Removed LNI 192.168.203.33@tcp [ 1825.285833] LNet: Removed LNI 192.168.203.33@tcp1 [ 1825.341299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1825.471437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1825.569298] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1826.948522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1827.001348] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1828.472208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 1828.578737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1828.631136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1828.704417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1828.776403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1828.854550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1828.942116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.33@tcp --health 0 [ 1829.074841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1829.472563] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.33@tcp [ 1829.480410] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1829.500060] LNet: Skipped 1 previous similar message [ 1844.075211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1844.342240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1844.390516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1844.445886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.33@tcp --health 500 [ 1844.714564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1851.714601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1851.870218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1851.923064] LNet: 75512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.941922] LNetError: 75512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.960039] LNetError: 75512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1851.991464] LNet: Removed LNI 192.168.203.33@tcp [ 1852.012959] LNet: Removed LNI 192.168.203.33@tcp1 [ 1852.046550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.174604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1852.250325] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1853.399174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1853.478685] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1855.292510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 1855.391263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1855.518692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1855.597312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1855.704605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1855.786693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1855.903502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1855.954565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.33@tcp --health 0 [ 1857.247387] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1857.250356] LNet: Skipped 3 previous similar messages [ 1857.622835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1868.593434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1868.654307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1868.709870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1877.425469] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:01:12 (1789567272) [ 1877.954588] LNet: 76570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.979016] LNetError: 76570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.984034] LNetError: 76570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1877.990972] LNet: Removed LNI 192.168.203.33@tcp [ 1878.002128] LNet: Removed LNI 192.168.203.33@tcp1 [ 1879.009228] Key type .llcrypt unregistered [ 1879.013345] Key type ._llcrypt unregistered [ 1887.154739] Key type ._llcrypt registered [ 1887.159336] Key type .llcrypt registered [ 1887.380051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1887.397709] alg: No test for adler32 (adler32-zlib) [ 1888.329626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1888.375118] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1899.336730] Lustre: DEBUG MARKER: Initial discovery [ 1899.411895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.133@tcp [ 1902.319615] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1902.579711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1902.670335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.133@tcp [ 1902.740397] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1902.777421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1902.849864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.133@tcp -r 1 -m GET -l 3 [ 1902.905824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.133@tcp -r 1 -m GET -e local_error [ 1902.936610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.133@tcp -r 1 -m PUT -l 6 [ 1903.797958] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 1904.774474] Lustre: DEBUG MARKER: Force oleg333-client.virtnet to discover 192.168.203.133@tcp (in background) [ 1905.385079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.133@tcp [ 1907.646905] Lustre: DEBUG MARKER: Force oleg333-server to discover 192.168.203.33@tcp [ 1908.513072] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1911.342517] Lustre: DEBUG MARKER: Wait for 77331 [ 1912.416256] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1913.304325] Lustre: DEBUG MARKER: Finished wait on 77331 [ 1913.457073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1913.551525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1913.938241] LNet: 77780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1913.952158] LNetError: 77780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1913.969881] LNet: Removed LNI 192.168.203.33@tcp [ 1914.690478] Key type .llcrypt unregistered [ 1914.696834] Key type ._llcrypt unregistered [ 1936.139492] Key type ._llcrypt registered [ 1936.145517] Key type .llcrypt registered [ 1937.981360] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:02:12 (1789567332) [ 1939.093381] Key type .llcrypt unregistered [ 1939.100824] Key type ._llcrypt unregistered [ 1948.029550] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1948.361473] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1948.380646] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1948.391212] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1951.248032] Key type ._llcrypt registered [ 1951.257650] Key type .llcrypt registered [ 1951.517157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1951.544562] alg: No test for adler32 (adler32-zlib) [ 1952.529785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1952.649227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1952.717474] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1953.904829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1953.933732] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1965.801906] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 10:02:39 (1789567359) [ 1966.169481] LNet: 79856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.191456] LNetError: 79856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1966.200409] LNet: Removed LNI 10.1.2.3@tcp [ 1966.247446] LNet: Removed LNI 192.168.203.33@tcp [ 1967.339225] Key type .llcrypt unregistered [ 1967.344933] Key type ._llcrypt unregistered [ 1978.175618] Key type ._llcrypt registered [ 1978.177513] Key type .llcrypt registered [ 1978.591349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1978.620968] alg: No test for adler32 (adler32-zlib) [ 1979.627438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1979.734423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1979.788339] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1981.058404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1981.100629] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1982.778413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.33@tcp [ 1983.905258] LNetError: Unexpected error -22 connecting to 192.168.203.33@tcp at host 192.168.203.33:988 [ 1991.611940] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 10:03:06 (1789567386) [ 1991.962358] LNet: 80984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.981532] LNetError: 80984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1992.005605] LNet: Removed LNI 10.1.2.3@tcp [ 1992.036239] LNet: Removed LNI 192.168.203.33@tcp [ 1992.942435] Key type .llcrypt unregistered [ 1992.947935] Key type ._llcrypt unregistered [ 2000.739461] Key type ._llcrypt registered [ 2000.741504] Key type .llcrypt registered [ 2001.053336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2001.077526] alg: No test for adler32 (adler32-zlib) [ 2002.029137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.072989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2002.103639] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 2003.012430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2003.031939] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2004.313757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.33@tcp1 --nid 192.168.203.33@tcp2 [ 2010.484246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp1 192.168.203.33@tcp2 [ 2010.507662] LNetError: 81819:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.33@tcp1 to 192.168.203.33@tcp2 - Route table may be misconfigured [ 2010.532662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp2 192.168.203.33@tcp1 [ 2020.682193] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 10:03:34 (1789567414) [ 2021.298460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2021.335880] LNet: 82393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2021.351585] LNetError: 82393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2022.367431] LNet: Removed LNI 192.168.203.33@tcp1 [ 2022.376787] LNetError: 82393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2022.409429] LNet: Removed LNI 192.168.203.33@tcp2 [ 2022.500648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2022.645819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2022.709513] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2024.136452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2024.187467] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 2025.817867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2025.892031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp 192.168.203.33@tcp [ 2025.924695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp1 192.168.203.33@tcp1 [ 2026.006503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.33@tcp -d 192.168.203.33@tcp -e network_timeout [ 2026.071350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -e network_timeout [ 2026.128514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -e network_timeout [ 2026.217463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -e network_timeout [ 2026.268687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2026.314650] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 2026.362783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2026.528509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2026.656116] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 2026.700463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2036.623894] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 10:03:50 (1789567430) [ 2036.898283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2036.929632] LNet: 83255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.948215] LNetError: 83255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2036.958069] LNet: Removed LNI 192.168.203.33@tcp [ 2037.997975] LNet: Removed LNI 192.168.203.33@tcp1 [ 2038.075899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2038.214908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 2038.698742] LNet: 83292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2039.349137] Key type .llcrypt unregistered [ 2039.350851] Key type ._llcrypt unregistered [ 2055.956335] Key type ._llcrypt registered [ 2055.957920] Key type .llcrypt registered [ 2058.010492] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 10:04:11 (1789567451) [ 2058.870232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2058.898165] alg: No test for adler32 (adler32-zlib) [ 2059.916629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.855402] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 2063.952280] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 10:04:18 (1789567458) [ 2064.342643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2064.406882] LNet: 84388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2064.435478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2064.532615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2064.618791] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2065.615269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2065.658466] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 2067.075841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2067.168752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp1 [ 2067.281677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp1 [ 2075.657120] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 10:04:29 (1789567469) [ 2078.572310] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg333-server [ 2081.525730] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 10:04:35 (1789567475) [ 2083.970648] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg333-server [ 2086.725983] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 10:04:40 (1789567480) [ 2089.247534] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg333-server [ 2092.333472] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 10:04:46 (1789567486) [ 2095.131367] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg333-server [ 2097.242560] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 10:04:51 (1789567491) [ 2100.100337] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg333-server [ 2102.293159] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 10:04:56 (1789567496) [ 2105.196482] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg333-server [ 2107.323410] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 10:05:02 (1789567502) [ 2109.698966] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg333-server [ 2111.754243] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 10:05:06 (1789567506) [ 2114.590971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg333-server [ 2116.775676] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 10:05:11 (1789567511) [ 2119.655745] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg333-server [ 2122.026802] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 10:05:16 (1789567516) [ 2122.254080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2122.311817] LNet: 88268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2122.318968] LNetError: 88268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2122.329760] LNet: Removed LNI 192.168.203.33@tcp [ 2123.372221] LNet: Removed LNI 192.168.203.33@tcp1 [ 2123.467637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.591582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2123.636273] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2124.642539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 2124.779642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2125.087665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2125.141074] LNet: 88417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.149947] LNetError: 88417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2125.159426] LNetError: 88417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2125.184689] LNet: Removed LNI 192.168.203.33@tcp [ 2125.229320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2125.327321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 2125.379075] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2126.481327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2126.742946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2126.795687] LNetError: 88562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2126.823185] LNet: Removed LNI 192.168.203.33@tcp [ 2126.851917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.916552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2126.946779] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2127.920737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 2128.026763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2128.255445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2128.299060] LNet: 88710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.313679] LNet: 88710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2129.382268] LNet: Removed LNI 192.168.203.33@tcp [ 2129.463499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.591816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 2129.650780] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2131.106224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2131.498710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.527674] LNetError: 88855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2131.533514] LNetError: 88855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2131.559176] LNet: Removed LNI 192.168.203.33@tcp [ 2131.612897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2131.697974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2131.722430] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2132.581881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 2132.680494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2134.180770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2134.217281] LNet: 89011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2134.223754] LNet: 89011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2135.271405] LNet: Removed LNI 192.168.203.33@tcp [ 2135.336251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.487567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 2135.534157] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2136.677446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2138.290652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2138.330822] LNetError: 89166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.338593] LNetError: 89166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2139.370903] LNet: Removed LNI 192.168.203.33@tcp [ 2139.400626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.528463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2139.587293] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2140.787560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 2140.924466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2141.269816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2142.378217] LNet: Removed LNI 192.168.203.33@tcp [ 2142.414087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2142.504080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 2142.557461] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2143.800899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2145.590263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2145.638979] LNet: 89468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2145.654996] LNet: 89468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2146.726349] LNet: Removed LNI 192.168.203.33@tcp [ 2146.793827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.910963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2146.981801] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2148.288771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 2148.373933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2148.815759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2148.887700] LNetError: 89615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.901970] LNetError: 89615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2149.996084] LNet: Removed LNI 192.168.203.33@tcp [ 2150.043800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.133471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 2150.185289] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2151.392975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2151.771900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2152.871809] LNet: Removed LNI 192.168.203.33@tcp [ 2152.904752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.029077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2153.077377] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2153.867686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 2154.038443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2154.446463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2155.562197] LNet: Removed LNI 192.168.203.33@tcp [ 2155.605830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.711543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 2155.743766] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2156.693659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2157.164944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2158.249192] LNet: Removed LNI 192.168.203.33@tcp [ 2158.310169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.389488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2158.475075] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2159.499193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 2159.595888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2159.648702] LNet: Removed LNI 192.168.203.33@tcp [ 2159.675680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.755497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2159.801968] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2160.849495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 2160.956056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2161.036983] LNet: Removed LNI 192.168.203.33@tcp [ 2161.077599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2161.230575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 2161.348654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2161.404076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2161.526434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 2161.698591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2161.753136] LNet: 90349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.757557] LNet: 90349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 2161.774447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2161.889043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 2162.017632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2162.088736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2162.169829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 2162.196147] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2163.179299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2163.230914] LNet: Removed LNI 192.168.203.33@tcp [ 2163.266917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2163.399378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2173.365751] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 10:06:07 (1789567567) [ 2173.696107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2173.775449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2173.857602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 2173.895371] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/180] [ 2175.553928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2175.591488] LNetError: 91214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2175.598860] LNetError: 91214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2175.629718] LNet: Removed LNI 192.168.203.33@tcp231 [ 2175.647555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2175.755356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2175.811563] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/180] [ 2175.823483] LNet: 91222:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2176.475376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2176.550512] LNet: Removed LNI 192.168.203.33@tcp231 [ 2176.578817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 2176.631348] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/180] [ 2178.455909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2178.527560] LNet: Removed LNI 192.168.203.33@tcp231 [ 2178.571560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2178.670191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2178.704990] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/0] [ 2178.711246] LNet: 91375:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2179.385892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2179.442401] LNet: Removed LNI 192.168.203.33@tcp231 [ 2179.459989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 2179.505035] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/0] [ 2181.298279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2181.340882] LNet: Removed LNI 192.168.203.33@tcp231 [ 2181.374351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.465472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2181.519625] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/60] [ 2181.536311] LNet: 91526:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2182.480300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2182.534582] LNet: Removed LNI 192.168.203.33@tcp231 [ 2182.577309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 2182.610067] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/60] [ 2184.241105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2184.281071] LNet: Removed LNI 192.168.203.33@tcp231 [ 2184.306421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2184.405439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2184.455886] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/180] [ 2184.464647] LNet: 91677:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2185.102498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2185.145733] LNet: Removed LNI 192.168.203.33@tcp231 [ 2185.182983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 2185.235805] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/180] [ 2196.330967] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 10:06:30 (1789567590) [ 2196.498953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2196.564941] LNet: 92382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.584349] LNet: 92382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 2196.617058] LNet: Removed LNI 192.168.203.33@tcp231 [ 2196.667991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2196.849182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 2196.915262] LNet: Added LNI 192.168.203.33@tcp232 [8/256/0/180] [ 2197.955585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2197.990333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2198.019250] LNet: Removed LNI 192.168.203.33@tcp232 [ 2198.049806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 2198.104682] LNet: Added LNI 192.168.203.33@tcp232 [8/256/0/180] [ 2199.433891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2199.502800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2199.543900] LNet: Removed LNI 192.168.203.33@tcp232 [ 2199.587581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 2199.642595] LNet: Added LNI 192.168.203.33@tcp232 [8/256/0/180] [ 2200.665709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2200.735864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2200.795940] LNet: Removed LNI 192.168.203.33@tcp232 [ 2200.830231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 2200.881753] LNet: Added LNI 192.168.203.33@tcp232 [8/256/0/180] [ 2202.009839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2202.086740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2202.149237] LNet: Removed LNI 192.168.203.33@tcp232 [ 2202.172213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 2211.162693] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 10:06:45 (1789567605) [ 2213.001673] Key type .llcrypt unregistered [ 2213.015304] Key type ._llcrypt unregistered [ 2221.056535] Key type ._llcrypt registered [ 2221.058261] Key type .llcrypt registered [ 2221.515869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2221.546987] alg: No test for adler32 (adler32-zlib) [ 2222.676791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2222.752322] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2232.101525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 2243.705215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.783995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2243.825766] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 2246.056285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_interrupt -r 2 [ 2246.114088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_interrupt -r 2 [ 2246.180848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_interrupt -r 2 [ 2246.232851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_interrupt -r 2 [ 2246.293973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_interrupt -r 2 [ 2246.351857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_interrupt -r 2 [ 2246.439771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_interrupt -r 2 [ 2246.552740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_interrupt -r 2 [ 2246.615676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp 192.168.203.133@tcp [ 2246.668591] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 2246.683455] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 2247.126718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2247.186350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2247.257515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2247.319495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2247.381242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2247.458298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2247.528857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2247.583748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2247.658837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2247.695773] LNet: 94180:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2247.746349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2247.826304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2247.916244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2247.966686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2248.019383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2248.092700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2248.963138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_dropped -r 2 [ 2249.051344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_dropped -r 2 [ 2249.117725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_dropped -r 2 [ 2249.203989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_dropped -r 2 [ 2249.268823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_dropped -r 2 [ 2249.337712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_dropped -r 2 [ 2249.420772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_dropped -r 2 [ 2249.490840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_dropped -r 2 [ 2249.547258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp 192.168.203.133@tcp [ 2249.566238] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 2250.042832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2250.111776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2250.177500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2250.227110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2250.316361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2250.405776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2250.485393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2250.568667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2250.623754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2250.667611] LNet: 94302:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2250.705745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2250.752638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2250.794209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2250.850589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2250.914209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2251.000633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2251.774373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_aborted -r 2 [ 2251.811549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_aborted -r 2 [ 2251.854941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_aborted -r 2 [ 2251.880810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_aborted -r 2 [ 2251.911527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_aborted -r 2 [ 2251.944480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_aborted -r 2 [ 2251.993100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_aborted -r 2 [ 2252.057996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_aborted -r 2 [ 2252.139264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp 192.168.203.133@tcp [ 2252.190310] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 2252.572199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2252.610433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2252.639396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2252.699385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2252.747271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2252.801510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2252.839202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2252.881529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2252.946817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2252.979769] LNet: 94424:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2253.000605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2253.055583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2253.109499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2253.159570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2253.217785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2253.301430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2254.113668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_no_route -r 2 [ 2254.213730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_no_route -r 2 [ 2254.303773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_no_route -r 2 [ 2254.403276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_no_route -r 2 [ 2254.480413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_no_route -r 2 [ 2254.544505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_no_route -r 2 [ 2254.629628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_no_route -r 2 [ 2254.747558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_no_route -r 2 [ 2254.826486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp 192.168.203.133@tcp [ 2254.880365] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 2255.344986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2255.415786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2255.486795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2255.574151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2255.652646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2255.710663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2255.782387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2255.850202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2255.897827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2255.922832] LNet: 94547:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2255.950935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2256.010724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2256.065647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2256.099969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2256.170745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2256.256873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2256.985210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_timeout -r 2 [ 2257.016771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_timeout -r 2 [ 2257.075584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_timeout -r 2 [ 2257.118765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_timeout -r 2 [ 2257.173658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_timeout -r 2 [ 2257.216962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_timeout -r 2 [ 2257.279899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_timeout -r 2 [ 2257.345327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_timeout -r 2 [ 2257.410968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp 192.168.203.133@tcp [ 2257.746731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2257.791875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2257.855163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2257.901823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2257.978040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2258.039522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2258.083419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2258.129431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2258.183972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2258.244848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2258.312342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2258.372951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2258.419521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2258.480675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2258.519974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2259.407161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e remote_dropped -r 2 [ 2259.470028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e remote_dropped -r 2 [ 2259.540377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e remote_dropped -r 2 [ 2259.613699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e remote_dropped -r 2 [ 2259.702272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e remote_dropped -r 2 [ 2259.768553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e remote_dropped -r 2 [ 2259.832863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e remote_dropped -r 2 [ 2259.916884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e remote_dropped -r 2 [ 2260.022593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp 192.168.203.133@tcp [ 2260.071062] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 2260.081862] LNet: Skipped 1 previous similar message [ 2260.095178] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.133@tcp [ 2260.458943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2260.507767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2260.557885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2260.647403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2260.717560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2260.773234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2260.850810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2260.894440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2260.955982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2260.975985] LNet: 94790:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2260.982949] LNet: 94790:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 2261.015319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2261.081914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2261.146381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2261.199537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2261.255800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2261.314641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2270.051633] LNet: 94866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2270.058823] LNetError: 94866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2270.066470] LNet: Removed LNI 192.168.203.33@tcp [ 2270.092230] LNet: Removed LNI 192.168.203.33@tcp1 [ 2270.629674] Key type .llcrypt unregistered [ 2270.635870] Key type ._llcrypt unregistered [ 2292.036909] Key type ._llcrypt registered [ 2292.039965] Key type .llcrypt registered [ 2293.919899] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 10:08:08 (1789567688) [ 2294.536559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2294.568477] alg: No test for adler32 (adler32-zlib) [ 2295.640953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.710931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2295.731787] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2297.144987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2297.211308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.33@tcp [ 2297.244477] LNetError: 95842:0:(api-ni.c:4493:lnet_ni_set_healthv()) manually adding local NI 192.168.203.33@tcp to recovery [ 2297.369498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.33@tcp [ 2297.533753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2307.221584] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 10:08:21 (1789567701) [ 2307.610468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2307.655403] LNet: 96431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2307.668870] LNetError: 96431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2307.687961] LNet: Removed LNI 192.168.203.33@tcp [ 2307.704802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2307.832508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2307.891903] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2309.019899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2309.046838] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 2310.565368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.33@tcp [ 2310.622479] LNetError: 96698:0:(api-ni.c:4493:lnet_ni_set_healthv()) manually adding local NI 192.168.203.33@tcp to recovery [ 2310.816944] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 2310.822807] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 2312.011677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.33@tcp [ 2322.553514] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 10:08:36 (1789567716) [ 2325.439362] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg333-server [ 2328.111630] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 10:08:42 (1789567722) [ 2330.220733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg333-server [ 2332.786293] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 10:08:47 (1789567727) [ 2335.166239] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg333-server [ 2337.677337] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:08:52 (1789567732) [ 2337.851973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2337.920309] LNet: 98316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.943602] LNetError: 98316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2337.960612] LNet: Removed LNI 192.168.203.33@tcp [ 2337.998538] LNet: Removed LNI 192.168.203.33@tcp1 [ 2338.154205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2338.334541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2338.389939] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2339.588102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2339.685569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2339.738857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2340.083158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2340.205473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2340.407652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2349.709175] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 10:09:04 (1789567744) [ 2350.579543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2350.620461] LNet: 99054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2350.634937] LNetError: 99054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2350.644556] LNetError: 99054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2350.697475] LNet: Removed LNI 192.168.203.33@tcp [ 2350.743542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.855272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2350.891028] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2360.907930] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:09:15 (1789567755) [ 2361.003030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2361.058554] LNet: 99750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2361.078566] LNetError: 99750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2361.110623] LNet: Removed LNI 192.168.203.33@tcp [ 2361.186782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2361.324942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2361.356951] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2372.200966] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:09:26 (1789567766) [ 2374.317688] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2377.103138] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:09:31 (1789567771) [ 2378.172223] LNet: 100793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2378.187439] LNetError: 100793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2378.255799] LNet: Removed LNI 192.168.203.33@tcp [ 2379.338911] Key type .llcrypt unregistered [ 2379.343203] Key type ._llcrypt unregistered [ 2389.618294] Key type ._llcrypt registered [ 2389.626504] Key type .llcrypt registered [ 2390.009879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2390.048960] alg: No test for adler32 (adler32-zlib) [ 2391.048803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2391.084067] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2400.798260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 2407.198111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.133@tcp [ 2407.990355] LNet: 101208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2407.996561] LNetError: 101208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2408.016178] LNet: Removed LNI 192.168.203.33@tcp [ 2408.681726] Key type .llcrypt unregistered [ 2408.686989] Key type ._llcrypt unregistered [ 2423.881229] Key type ._llcrypt registered [ 2423.886153] Key type .llcrypt registered [ 2426.021918] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2428.269071] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2430.558704] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:10:24 (1789567824) [ 2432.899986] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2435.424447] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:10:29 (1789567829) [ 2438.258610] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg333-server [ 2440.373311] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 10:10:34 (1789567834) [ 2442.936405] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg333-server [ 2445.267444] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:10:39 (1789567839) [ 2447.610241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2447.897675] alg: No test for adler32 (adler32-zlib) [ 2449.189440] Lustre: Lustre: Build Version: 2.17.58_40_g81e4dae [ 2449.632904] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2451.375564] Key type lgssc registered [ 2453.015071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2480.472311] Key type lgssc unregistered [ 2480.910909] LNet: 104383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2480.932267] LNetError: 104383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2480.962954] LNet: Removed LNI 192.168.203.33@tcp [ 2481.783252] Key type .llcrypt unregistered [ 2481.788303] Key type ._llcrypt unregistered [ 2499.808641] Key type ._llcrypt registered [ 2499.812984] Key type .llcrypt registered [ 2501.558858] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 10:11:35 (1789567895) [ 2502.051887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2502.082146] alg: No test for adler32 (adler32-zlib) [ 2503.050797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2503.163252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2503.226902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2503.286899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2503.371198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2512.394063] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 10:11:46 (1789567906) [ 2512.556807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2512.592373] LNet: 105789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2512.629674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2512.722532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2512.776961] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2514.157125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2514.215334] LNet: 105940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2514.227559] LNetError: 105940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2514.248824] LNet: Removed LNI 192.168.203.33@tcp [ 2514.299413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2514.416482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2514.460360] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2515.246983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2515.281342] LNet: 105968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2515.287698] LNetError: 105968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2515.307149] LNet: Removed LNI 192.168.203.33@tcp [ 2515.336968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2515.442882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2515.477896] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2525.153629] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 10:11:59 (1789567919) [ 2525.455413] LNet: 106557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2525.496914] LNetError: 106557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2525.523838] LNet: Removed LNI 192.168.203.33@tcp [ 2526.263180] Key type .llcrypt unregistered [ 2526.270568] Key type ._llcrypt unregistered [ 2536.895969] Key type ._llcrypt registered [ 2536.898352] Key type .llcrypt registered [ 2537.130912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2537.184750] alg: No test for adler32 (adler32-zlib) [ 2538.301341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2538.440676] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2538.444786] LNet: *** cfs_fail_loc=e003, val=100*** [ 2539.470872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2539.495771] LNet: *** cfs_fail_loc=e003, val=100*** [ 2539.502636] LNet: Skipped 2 previous similar messages [ 2539.512648] Ping reply received with mapping: [ 2539.517967] found metadata reply NI[192.168.203.33@tcp] [ 2539.521213] 21 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2539.525816] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2539.532614] 5a 5a 5a 5a 5a 5a 5a 5a [ 2539.723810] LNet: 106955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2539.736982] LNetError: 106955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2540.782239] LNet: Removed LNI 192.168.203.33@tcp [ 2541.774467] Key type .llcrypt unregistered [ 2541.776711] Key type ._llcrypt unregistered [ 2557.084474] Key type ._llcrypt registered [ 2557.088321] Key type .llcrypt registered [ 2558.906038] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:12:33 (1789567953) [ 2559.593641] Key type .llcrypt unregistered [ 2559.598681] Key type ._llcrypt unregistered [ 2567.813405] Key type ._llcrypt registered [ 2567.815571] Key type .llcrypt registered [ 2568.148768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2568.193859] alg: No test for adler32 (adler32-zlib) [ 2569.531481] LNetError: 107992:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2570.260649] Key type .llcrypt unregistered [ 2570.266482] Key type ._llcrypt unregistered [ 2578.530240] Key type ._llcrypt registered [ 2578.537370] Key type .llcrypt registered [ 2580.811380] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 10:12:54 (1789567974) [ 2581.585708] Key type .llcrypt unregistered [ 2581.592529] Key type ._llcrypt unregistered [ 2590.064994] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2590.072439] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2592.964120] Key type ._llcrypt registered [ 2592.967533] Key type .llcrypt registered [ 2593.448956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2593.492632] alg: No test for adler32 (adler32-zlib) [ 2594.608885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2594.734317] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2609.458984] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 10:13:23 (1789568003) [ 2609.676650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2609.717464] LNet: 109768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2609.754306] LNetError: 109768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2609.805441] LNet: Removed LNI 10.1.2.3@tcp [ 2609.857494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2611.101130] Key type .llcrypt unregistered [ 2611.102898] Key type ._llcrypt unregistered [ 2621.119503] Key type ._llcrypt registered [ 2621.121683] Key type .llcrypt registered [ 2621.533121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2621.572904] alg: No test for adler32 (adler32-zlib) [ 2622.808875] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2623.001261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2623.351801] LNet: 110054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2623.359578] LNetError: 110054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2623.385781] LNet: Removed LNI 192.168.203.33@tcp [ 2624.162575] Key type .llcrypt unregistered [ 2624.168954] Key type ._llcrypt unregistered [ 2640.070748] Key type ._llcrypt registered [ 2640.072590] Key type .llcrypt registered [ 2642.389425] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 10:13:56 (1789568036) [ 2643.283789] Key type .llcrypt unregistered [ 2643.290021] Key type ._llcrypt unregistered [ 2651.217609] Key type ._llcrypt registered [ 2651.226460] Key type .llcrypt registered [ 2651.601609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2651.652281] alg: No test for adler32 (adler32-zlib) [ 2661.105732] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 10:14:15 (1789568055) [ 2661.215394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2661.321185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2661.440326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2661.478443] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2662.593471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2662.670209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2662.704045] LNet: Removed LNI 192.168.203.33@tcp [ 2663.079135] LNet: 111794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2663.752428] Key type .llcrypt unregistered [ 2663.762799] Key type ._llcrypt unregistered [ 2677.394786] Key type ._llcrypt registered [ 2677.401480] Key type .llcrypt registered [ 2679.375795] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 10:14:33 (1789568073) [ 2679.800721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2679.816761] alg: No test for adler32 (adler32-zlib) [ 2680.890774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2680.969569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2681.007157] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2682.086656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2682.141732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.33@tcp [ 2682.186836] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2682.196607] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2682.203138] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2682, 0, 0 [ 2682.212890] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2682.218264] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2682.225806] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2682.233921] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2682, 0, 0 [ 2682.243343] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2682.248631] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2682.259313] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2682.263877] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2682, 0, 2732 [ 2682.278338] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2682.285382] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2682.293621] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2682.306733] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2682, 0, 2732 [ 2682.319712] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2690.912441] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 10:14:45 (1789568085) [ 2691.530813] LNet: 113331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2691.539447] LNetError: 113331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2691.556026] LNet: Removed LNI 192.168.203.33@tcp [ 2692.373865] Key type .llcrypt unregistered [ 2692.375956] Key type ._llcrypt unregistered [ 2700.345478] Key type ._llcrypt registered [ 2700.347384] Key type .llcrypt registered [ 2700.736621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2700.787810] alg: No test for adler32 (adler32-zlib) [ 2701.890483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2701.945125] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2712.941864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 2725.545207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2725.661812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2725.714589] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 2727.263361] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2727.561103] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2727.569240] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2727.599145] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2730.104817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2730.143025] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2744.031559] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 10:15:38 (1789568138) [ 2744.379924] LNet: 114852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2744.400369] LNetError: 114852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2745.439228] LNet: Removed LNI 192.168.203.33@tcp [ 2745.443709] LNetError: 114852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2745.458411] LNet: Removed LNI 192.168.203.33@tcp1 [ 2745.472621] LNet: Removed LNI 10.1.2.3@tcp99 [ 2746.549853] Key type .llcrypt unregistered [ 2746.556841] Key type ._llcrypt unregistered [ 2754.492545] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2754.497258] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2757.324161] Key type ._llcrypt registered [ 2757.327855] Key type .llcrypt registered [ 2757.760725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2757.783833] alg: No test for adler32 (adler32-zlib) [ 2758.892691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2758.996164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2759.055472] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2760.145659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2760.183973] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2763.433889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2763.516166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2763.624207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2763.659874] LNetError: 115595:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2763.729736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2763.822905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2764.649986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2764.751183] LNet: 115615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2764.765230] LNetError: 115615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2764.774184] LNet: Removed LNI 10.1.2.3@tcp [ 2764.800242] LNet: Removed LNI 192.168.203.33@tcp [ 2765.832565] Key type .llcrypt unregistered [ 2765.837648] Key type ._llcrypt unregistered [ 2783.503779] Key type ._llcrypt registered [ 2783.505945] Key type .llcrypt registered [ 2785.316214] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 10:16:19 (1789568179) [ 2786.322990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2786.345624] alg: No test for adler32 (adler32-zlib) [ 2787.478303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2787.564513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2787.612287] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2788.721530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2788.880388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg333-client@tcp [ 2797.623615] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 10:16:32 (1789568192) [ 2798.104877] LNet: 117149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2798.123343] LNetError: 117149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2798.152623] LNet: Removed LNI 192.168.203.33@tcp [ 2799.238627] Key type .llcrypt unregistered [ 2799.243244] Key type ._llcrypt unregistered [ 2807.136483] Key type ._llcrypt registered [ 2807.151390] Key type .llcrypt registered [ 2807.482670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2807.523319] alg: No test for adler32 (adler32-zlib) [ 2808.463644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2808.502027] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2808.538689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2817.383706] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 10:16:51 (1789568211) [ 2819.630254] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2822.384688] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 10:16:56 (1789568216) [ 2824.435621] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2826.941721] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 10:17:01 (1789568221) [ 2827.058857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2827.104079] LNet: 118565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2827.115087] LNetError: 118565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2827.134316] LNet: Removed LNI 192.168.203.33@tcp [ 2827.179869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2827.290641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2827.339111] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2828.124341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2828.232825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2837.850326] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 10:17:12 (1789568232) [ 2838.069977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2838.140653] LNet: 119281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2838.144257] LNetError: 119281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2839.206947] LNet: Removed LNI 192.168.203.33@tcp [ 2839.261301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2839.386766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2839.430833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2839.815256] LNet: 119319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2840.571691] Key type .llcrypt unregistered [ 2840.585256] Key type ._llcrypt unregistered [ 2856.772961] Key type ._llcrypt registered [ 2856.778803] Key type .llcrypt registered [ 2858.706737] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 10:17:32 (1789568252) [ 2859.139363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2859.157844] alg: No test for adler32 (adler32-zlib) [ 2860.149713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2860.245255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2860.268840] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2861.266511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2861.352796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2869.644707] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 10:17:44 (1789568264) [ 2869.777086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2869.825058] LNet: 120833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2869.831976] LNetError: 120833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2869.849381] LNet: Removed LNI 192.168.203.33@tcp [ 2869.887642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2870.051510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2870.105686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2878.488639] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 10:17:53 (1789568273) [ 2878.628827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2878.652247] LNet: 121412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2878.694835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2878.835671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2879.488392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2879.539378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2879.615228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2880.106373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2880.160320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2880.307490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2880.341457] LNet: 121453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2880.399679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2880.567425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2881.489336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2881.555315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2881.653975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2882.380627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2882.415214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2890.956743] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 10:18:05 (1789568285) [ 2891.078532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2891.130685] LNet: 122048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2891.176914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2891.346652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2891.380638] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2892.719246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2892.777105] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2893.757987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2893.838810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2894.165625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2894.187696] LNetError: 122333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2895.202820] LNet: Removed LNI 192.168.203.33@tcp [ 2895.208425] LNetError: 122333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2895.234905] LNet: Removed LNI 192.168.203.33@tcp2 [ 2895.324915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2895.515561] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2895.520287] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2895.527397] LNet: 122339:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2896.588330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2897.034870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2897.052439] LNet: 122377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2897.056986] LNet: 122377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2898.079518] LNet: Removed LNI 192.168.203.33@tcp [ 2898.092468] LNetError: 122377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2898.108114] LNetError: 122377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2898.132866] LNet: Removed LNI 192.168.203.33@tcp2 [ 2898.214660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2898.336769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2898.382217] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2899.473912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2899.512739] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2900.628718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2900.669667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2900.745965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2901.011593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2902.048963] LNet: Removed LNI 192.168.203.33@tcp [ 2903.081173] LNet: Removed LNI 192.168.203.33@tcp2 [ 2903.134240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2903.219850] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2903.224895] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2903.230883] LNet: 122668:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2904.179714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2904.565104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2904.609806] LNetError: 122705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2904.615840] LNetError: 122705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2904.630162] LNet: Removed LNI 192.168.203.33@tcp [ 2905.704286] LNet: Removed LNI 192.168.203.33@tcp2 [ 2905.748074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2905.866289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2905.913619] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2906.940741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2906.978419] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2908.405133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2908.512198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2908.945574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2909.004053] LNet: 122985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2909.011564] LNet: 122985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2910.048358] LNet: Removed LNI 192.168.203.33@tcp [ 2910.071729] LNet: Removed LNI 192.168.203.33@tcp2 [ 2910.148461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2910.246696] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2910.251380] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2910.256647] LNet: 122991:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2911.055806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2911.368696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2912.415509] LNet: Removed LNI 192.168.203.33@tcp [ 2912.428912] LNet: Removed LNI 192.168.203.33@tcp2 [ 2912.506248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2912.622803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2912.726994] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2914.110332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2914.145636] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2915.432825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2915.499520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2915.631757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2916.135214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2916.167326] LNetError: 123313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2916.176606] LNetError: 123313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2916.184725] LNet: Removed LNI 192.168.203.33@tcp [ 2917.221623] LNet: Removed LNI 192.168.203.33@tcp2 [ 2917.292170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2917.403392] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2917.412166] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2917.418379] LNet: 123320:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2917.792330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2918.044381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2918.085322] LNet: Removed LNI 192.168.203.33@tcp [ 2919.152427] LNet: Removed LNI 192.168.203.33@tcp2 [ 2919.195444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2919.299259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2919.343604] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2920.576273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2920.599990] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2921.725720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2921.803626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2921.919191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2922.606334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2923.679872] LNet: Removed LNI 192.168.203.33@tcp [ 2923.695741] LNet: Removed LNI 192.168.203.33@tcp2 [ 2923.742369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2923.885684] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2923.893912] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2923.904658] LNet: 123660:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2924.804684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2925.308825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2925.367772] LNet: 123712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2925.372872] LNet: 123712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2926.432631] LNet: Removed LNI 192.168.203.33@tcp [ 2926.453796] LNet: Removed LNI 192.168.203.33@tcp2 [ 2926.495723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2926.569390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2926.602739] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2927.349106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2927.410275] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2928.627816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2928.684912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2928.719939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2928.806595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2929.262524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2930.335346] LNet: Removed LNI 192.168.203.33@tcp [ 2931.365800] LNet: Removed LNI 192.168.203.33@tcp2 [ 2931.415477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2931.542256] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2931.556471] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2932.500865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2941.474238] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2943.477560] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 10:18:58 (1789568338) [ 2943.575168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2943.612470] LNetError: 124773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2943.626677] LNetError: 124773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2943.633188] LNet: Removed LNI 192.168.203.33@tcp [ 2944.678634] LNet: Removed LNI 192.168.203.33@tcp2 [ 2944.716661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2945.186162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2945.230748] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2946.163419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2946.184707] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2946.985975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2947.003624] LNet: Removed LNI 10.1.2.3@tcp [ 2947.018713] LNet: Removed LNI 192.168.203.33@tcp [ 2947.049229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2947.157701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2947.203659] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2947.212676] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2947.602270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2947.623684] LNet: Removed LNI 192.168.203.33@tcp [ 2947.644941] LNet: Removed LNI 10.1.2.3@tcp [ 2955.762466] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 10:19:10 (1789568350) [ 2955.890946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2955.991867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2956.130417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2956.172309] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2957.196907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2957.226882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2957.287379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2957.317076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2957.346348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2957.379200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2957.419910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2957.465842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2957.516700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2957.565899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2957.618561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2957.665491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2957.862295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2966.286888] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 10:19:20 (1789568360) [ 2966.646378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2966.707907] LNet: 126397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2966.720921] LNet: 126397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2967.783625] LNet: Removed LNI 192.168.203.33@tcp [ 2967.836467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2967.920368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2967.964837] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2969.081523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2969.110457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2969.175061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2969.207833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2969.240966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2969.276211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2969.312646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2969.390966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2969.448141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2969.489630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2969.561824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2969.645836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2969.863996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2969.936844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2970.009213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2970.091823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2970.149847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2970.209921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2970.241840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2970.281611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2970.357262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2970.400874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2970.445470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2979.645395] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 10:19:33 (1789568373) [ 2979.931804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2980.007685] LNetError: 127153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2980.035381] LNetError: 127153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2980.081973] LNet: Removed LNI 192.168.203.33@tcp [ 2980.106219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2980.382165] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2980.683620] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2980.711180] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2980.734066] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2983.384061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2983.456531] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2985.087137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2985.128347] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2988.954687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2989.055812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2989.158872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.33@tcp -m PUT -e local_timeout -r 1 [ 2989.303471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2989.407514] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2989.420702] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2989.734206] LNetError: 127219:0:(lib-move.c:4794:lnet_parse()) 192.168.203.33@tcp, src 192.168.203.33@tcp: Dropping PUT (error -108 looking up sender) [ 2990.754111] LNet: Removed LNI 10.1.2.3@tcp [ 2991.784890] LNet: Removed LNI 192.168.203.33@tcp [ 2992.884248] Key type .llcrypt unregistered [ 2992.899492] Key type ._llcrypt unregistered [ 3011.764541] Key type ._llcrypt registered [ 3011.769756] Key type .llcrypt registered [ 3013.741181] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 10:20:07 (1789568407) [ 3014.927396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3015.001283] alg: No test for adler32 (adler32-zlib) [ 3016.068847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3016.743496] LNet: 128530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3017.694937] Key type .llcrypt unregistered [ 3017.698892] Key type ._llcrypt unregistered [ 3025.357173] Key type ._llcrypt registered [ 3025.359255] Key type .llcrypt registered [ 3025.596045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3025.625603] alg: No test for adler32 (adler32-zlib) [ 3026.654693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3026.700975] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 3037.443524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 3037.700638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 3038.703469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.133@tcp [ 3039.010296] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.133@tcp [ 3039.214424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3039.324951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3039.431497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3039.541934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3039.676038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3039.773312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3039.851929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3039.944274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3040.016041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3040.070144] LNet: 128981:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3040.126103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3040.208720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3040.257141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3040.290651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3040.336775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3040.424262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3041.264359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3041.538107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3041.576940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3041.602859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3041.634705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3041.681532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3041.726030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3041.749338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3041.782696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3041.816267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3041.829754] LNet: 129100:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3041.846290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3041.908623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3041.963394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3042.011640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3042.049277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3042.102443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3043.132623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.133@tcp [ 3043.528213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3043.569917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3043.622106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3043.667837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3043.723107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3043.766596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3043.823483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3043.922734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3044.028435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3044.076473] LNet: 129220:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3044.137450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3044.243625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3044.347525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3044.424618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3044.501849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3044.559252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3046.124908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3046.569861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3046.610209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3046.636594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3046.666989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3046.701495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3046.742708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3046.799255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3046.825958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3046.857891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3046.887965] LNet: 129339:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3046.908357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3046.927861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3046.957453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3046.983212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3047.017198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3047.059656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3047.433553] LNet: 129385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3047.440296] LNetError: 129385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3047.454845] LNet: Removed LNI 192.168.203.33@tcp [ 3048.049800] Key type .llcrypt unregistered [ 3048.052638] Key type ._llcrypt unregistered [ 3071.394485] Key type ._llcrypt registered [ 3071.396623] Key type .llcrypt registered [ 3072.878810] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:21:07 (1789568467) [ 3073.427397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3073.460370] alg: No test for adler32 (adler32-zlib) [ 3074.541383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3075.212166] LNet: 130272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3076.272361] Key type .llcrypt unregistered [ 3076.275035] Key type ._llcrypt unregistered [ 3083.811709] Key type ._llcrypt registered [ 3083.816440] Key type .llcrypt registered [ 3084.086902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3084.106926] alg: No test for adler32 (adler32-zlib) [ 3085.128822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3085.181628] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 3093.462669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 3103.845467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3103.940719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3103.987760] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 3105.478470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 3106.484947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.133@tcp [ 3106.722170] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.133@tcp [ 3106.947451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3107.001784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3107.071182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3107.123361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3107.169660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3107.212514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3107.267912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3107.316073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3107.377442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3107.407751] LNet: 130925:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3107.455853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3107.513380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3107.571717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3107.597937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3107.658715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3107.718066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3108.769671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3109.141794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3109.176298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3109.202550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3109.235428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3109.265712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3109.298561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3109.341341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3109.388850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3109.438408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3109.469897] LNet: 131044:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3109.499283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3109.557819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3109.617965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3109.708389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3109.786886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3109.850737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3110.494576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.133@tcp [ 3110.710560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3110.765508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3110.813664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3110.869919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3110.940457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3111.011611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3111.083378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3111.112928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3111.146623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3111.190303] LNet: 131164:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3111.213501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3111.276148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3111.331882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3111.407469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3111.476673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3111.512750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3112.398344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3112.832446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3112.868535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3112.926210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3112.989282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3113.019304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3113.067558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3113.099489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3113.173933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3113.223977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3113.248316] LNet: 131283:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3113.284868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3113.325679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3113.363735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3113.398770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3113.448769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3113.510989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3119.705916] LNet: 131364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3119.717102] LNetError: 131364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3119.727121] LNet: Removed LNI 192.168.203.33@tcp [ 3119.756850] LNet: Removed LNI 192.168.203.33@tcp1 [ 3120.181695] Key type .llcrypt unregistered [ 3120.184535] Key type ._llcrypt unregistered [ 3142.849299] Key type ._llcrypt registered [ 3142.860687] Key type .llcrypt registered [ 3144.468838] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 10:22:18 (1789568538) [ 3145.040812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3145.078281] alg: No test for adler32 (adler32-zlib) [ 3146.252455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3146.413972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3146.484564] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 3147.715899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3147.759840] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 3149.040854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 3149.150073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 3150.239494] LNet: Removed LNI 192.168.203.33@tcp [ 3158.552465] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 10:22:33 (1789568553) [ 3160.376811] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 3162.614572] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 10:22:37 (1789568557) [ 3172.645273] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 10:22:47 (1789568567) [ 3182.652743] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 10:22:57 (1789568577) [ 3192.885965] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 10:23:07 (1789568587) [ 3202.524861] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 10:23:17 (1789568597) [ 3212.928689] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 10:23:27 (1789568607) [ 3224.432267] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 10:23:38 (1789568618) [ 3237.190694] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 10:23:51 (1789568631) [ 3249.004418] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 10:24:02 (1789568642) [ 3258.979906] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 10:24:13 (1789568653) [ 3267.903542] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 10:24:22 (1789568662) [ 3277.137752] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 10:24:31 (1789568671) [ 3286.352496] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 10:24:40 (1789568680) [ 3295.910793] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 10:24:50 (1789568690) [ 3305.645703] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 10:25:00 (1789568700) [ 3315.898800] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 10:25:10 (1789568710) [ 3325.643529] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 10:25:20 (1789568720) [ 3335.893635] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 10:25:30 (1789568730) [ 3336.030601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3336.062277] LNet: 143058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3336.077805] LNetError: 143058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3336.113236] LNet: Removed LNI 192.168.203.33@tcp2 [ 3336.152447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3336.439976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3336.487549] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 3338.009915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3338.030743] LNet: 143208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3338.039356] LNetError: 143208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3338.085215] LNet: Removed LNI 192.168.203.33@tcp [ 3338.115335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3338.216197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 3338.252561] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 3347.645346] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 10:25:42 (1789568742) [ 3358.052596] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 10:25:52 (1789568752) [ 3367.805512] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 10:26:02 (1789568762) [ 3376.770634] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:26:11 (1789568771) [ 3384.729402] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:26:19 (1789568779) [ 3393.590324] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:26:28 (1789568788) [ 3402.635359] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:26:37 (1789568797) [ 3411.389304] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:26:46 (1789568806) [ 3422.077672] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:26:56 (1789568816) [ 3431.732827] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:27:06 (1789568826) [ 3441.483916] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:27:15 (1789568835) [ 3450.028560] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:27:24 (1789568844) [ 3458.889241] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 3218 sec ========== 10:27:33 (1789568853) [ 3459.262152] LNet: 150686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3459.268572] LNetError: 150686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3459.294751] LNet: Removed LNI 192.168.203.33@tcp [ 3460.099228] Key type .llcrypt unregistered [ 3460.104971] Key type ._llcrypt unregistered [ 3475.584490] Key type ._llcrypt registered [ 3475.587267] Key type .llcrypt registered [ 3476.153631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3476.193294] alg: No test for adler32 (adler32-zlib) [ 3477.491690] Lustre: Lustre: Build Version: 2.17.58_40_g81e4dae [ 3477.970643] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 3479.751242] Key type lgssc registered [ 3481.672928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3565.824751] Lustre: Mounted lustre-client - version 2.17.58_40_g81e4dae [ 3572.067220] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3582.291848] Lustre: Unmounted lustre-client [ 3615.795371] Key type lgssc unregistered [ 3616.180173] LNet: 153639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3616.187797] LNetError: 153639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3616.201256] LNet: Removed LNI 192.168.203.33@tcp [ 3617.058354] Key type .llcrypt unregistered [ 3617.061893] Key type ._llcrypt unregistered