[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 587339746 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002316] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011096] LSM: Security Framework initializing [ 0.012050] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.022323] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027313] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030116] *** VALIDATE tmpfs *** [ 0.031482] *** VALIDATE proc *** [ 0.033202] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.036225] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043393] debug: unmapping init [mem 0xffffffffbd459000-0xffffffffbd460fff] [ 0.045180] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046685] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047022] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052011] ... fixed-purpose events: 3 [ 0.053009] ... event mask: 000000070000000f [ 0.054277] rcu: Hierarchical SRCU implementation. [ 0.056600] smp: Bringing up secondary CPUs ... [ 0.057639] x86: Booting SMP configuration: [ 0.058025] .... node #0, CPUs: #1 #2 #3 [ 0.062362] smp: Brought up 1 node, 4 CPUs [ 0.064010] smpboot: Max logical packages: 1 [ 0.065013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136082] node 0 deferred pages initialised in 67ms [ 0.140012] devtmpfs: initialized [ 0.141239] x86/mm: Memory block size: 128MB [ 0.143783] gcov: version magic: 0x41383552 [ 0.145228] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146064] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149450] pinctrl core: initialized pinctrl subsystem [ 0.151221] [ 0.151713] ************************************************************* [ 0.154012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156008] ** ** [ 0.158010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160013] ** ** [ 0.162027] ** This means that this kernel is built to expose internal ** [ 0.165012] ** IOMMU data structures, which may compromise security on ** [ 0.167012] ** your system. ** [ 0.170011] ** ** [ 0.171006] ** If you see this message and you are not debugging the ** [ 0.174010] ** kernel, report this immediately to your vendor! ** [ 0.176022] ** ** [ 0.179012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181011] ************************************************************* [ 0.183941] NET: Registered protocol family 16 [ 0.186522] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193094] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.198014] cpuidle: using governor menu [ 0.199592] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202575] PCI: Using configuration type 1 for base access [ 0.205184] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217067] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224159] cryptd: max_cpu_qlen set to 1000 [ 0.227379] ACPI: Added _OSI(Module Device) [ 0.229022] ACPI: Added _OSI(Processor Device) [ 0.231014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.233017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237740] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.244875] ACPI: Interpreter enabled [ 0.247099] ACPI: PM: (supports S0 S3 S4 S5) [ 0.249013] ACPI: Using IOAPIC for interrupt routing [ 0.250122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.254400] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.267046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.278338] acpiphp: Slot [2] registered [ 0.280168] acpiphp: Slot [5] registered [ 0.281148] acpiphp: Slot [6] registered [ 0.282104] acpiphp: Slot [3] registered [ 0.284134] acpiphp: Slot [4] registered [ 0.285106] acpiphp: Slot [7] registered [ 0.287107] acpiphp: Slot [8] registered [ 0.289123] acpiphp: Slot [9] registered [ 0.290115] acpiphp: Slot [10] registered [ 0.292098] acpiphp: Slot [11] registered [ 0.293093] acpiphp: Slot [12] registered [ 0.295085] acpiphp: Slot [13] registered [ 0.296131] acpiphp: Slot [14] registered [ 0.297081] acpiphp: Slot [15] registered [ 0.299122] acpiphp: Slot [16] registered [ 0.300066] acpiphp: Slot [17] registered [ 0.302097] acpiphp: Slot [18] registered [ 0.303086] acpiphp: Slot [19] registered [ 0.304102] acpiphp: Slot [20] registered [ 0.306077] acpiphp: Slot [21] registered [ 0.307112] acpiphp: Slot [22] registered [ 0.308318] acpiphp: Slot [23] registered [ 0.310288] acpiphp: Slot [24] registered [ 0.313168] acpiphp: Slot [25] registered [ 0.315151] acpiphp: Slot [26] registered [ 0.316254] acpiphp: Slot [27] registered [ 0.318150] acpiphp: Slot [28] registered [ 0.320161] acpiphp: Slot [29] registered [ 0.322151] acpiphp: Slot [30] registered [ 0.324155] acpiphp: Slot [31] registered [ 0.325174] PCI host bridge to bus 0000:00 [ 0.327032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.329039] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.332040] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.336039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.339053] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.342039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345227] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.349014] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352397] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.363014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.368463] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.371024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.379553] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.383101] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387067] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.391062] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.397020] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.409881] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.414818] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.422078] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.431014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.441018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.460015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.470336] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.477022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.483023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.501022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.512232] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.515491] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.518456] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.521419] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.523301] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.529133] iommu: Default domain type: Passthrough [ 0.532545] SCSI subsystem initialized [ 0.533202] ACPI: bus type USB registered [ 0.535165] usbcore: registered new interface driver usbfs [ 0.537211] usbcore: registered new interface driver hub [ 0.540131] usbcore: registered new device driver usb [ 0.542302] pps_core: LinuxPPS API ver. 1 registered [ 0.545018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.548118] PTP clock support registered [ 0.550077] EDAC MC: Ver: 3.0.0 [ 0.551346] PCI: Using ACPI for IRQ routing [ 0.552945] NetLabel: Initializing [ 0.553019] NetLabel: domain hash size = 128 [ 0.554017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.555173] NetLabel: unlabeled traffic allowed by default [ 0.559067] vgaarb: loaded [ 0.561010] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.562024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.568533] clocksource: Switched to clocksource kvm-clock [ 0.672239] VFS: Disk quotas dquot_6.6.0 [ 0.673341] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.674899] *** VALIDATE ramfs *** [ 0.675700] *** VALIDATE hugetlbfs *** [ 0.676926] pnp: PnP ACPI init [ 0.678479] pnp: PnP ACPI: found 6 devices [ 0.694748] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.698281] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.700726] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.702976] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.705634] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.708142] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.711273] NET: Registered protocol family 2 [ 0.714645] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.721357] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.726094] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.732334] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.735663] TCP: Hash tables configured (established 65536 bind 65536) [ 0.737544] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.739376] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.741147] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.744726] NET: Registered protocol family 1 [ 0.746982] RPC: Registered named UNIX socket transport module. [ 0.748965] RPC: Registered udp transport module. [ 0.750843] RPC: Registered tcp transport module. [ 0.752311] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.754089] NET: Registered protocol family 44 [ 0.756142] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.759266] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.762309] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.765059] PCI: CLS 0 bytes, default 64 [ 0.766942] Unpacking initramfs... [ 2.195183] debug: unmapping init [mem 0xffff8bf0bcc64000-0xffff8bf0bffcffff] [ 2.199416] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.202241] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.205616] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.699620] Initialise system trusted keyrings [ 2.701381] Key type blacklist registered [ 2.703532] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.714850] zbud: loaded [ 2.717922] *** VALIDATE nfs *** [ 2.719431] *** VALIDATE nfs4 *** [ 2.721248] pstore: using deflate compression [ 2.725887] Platform Keyring initialized [ 2.833089] NET: Registered protocol family 38 [ 2.835333] Key type asymmetric registered [ 2.836931] Asymmetric key parser 'x509' registered [ 2.838307] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.841776] io scheduler mq-deadline registered [ 2.844103] io scheduler kyber registered [ 2.847247] io scheduler bfq registered [ 2.849621] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.852784] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.856056] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.859366] ACPI: Power Button [PWRF] [ 2.864420] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.869666] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.882092] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.912449] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.941885] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.946875] Non-volatile memory driver v1.3 [ 2.948926] Linux agpgart interface v0.103 [ 2.978621] virtio_blk virtio1: [vda] 146944 512-byte logical blocks (75.2 MB/71.8 MiB) [ 2.981817] vda: detected capacity change from 0 to 75235328 [ 2.994685] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.997282] vdb: detected capacity change from 0 to 1073741824 [ 3.002678] libphy: Fixed MDIO Bus: probed [ 3.007649] usbcore: registered new interface driver usbserial_generic [ 3.010239] usbserial: USB Serial support registered for generic [ 3.012827] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.017077] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.018678] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.020513] mousedev: PS/2 mouse device common for all mice [ 3.022871] rtc_cmos 00:05: RTC can wake from S4 [ 3.025869] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.029769] rtc_cmos 00:05: registered as rtc0 [ 3.032862] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.033562] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.036349] intel_pstate: CPU model not supported [ 3.041192] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.046465] hid: raw HID events driver (C) Jiri Kosina [ 3.049159] usbcore: registered new interface driver usbhid [ 3.053199] usbhid: USB HID core driver [ 3.055519] drop_monitor: Initializing network drop monitor service [ 3.057561] Initializing XFRM netlink socket [ 3.059525] NET: Registered protocol family 10 [ 3.062310] Segment Routing with IPv6 [ 3.063939] NET: Registered protocol family 17 [ 3.066211] mpls_gso: MPLS GSO support [ 3.071307] RAS: Correctable Errors collector initialized. [ 3.073705] AVX version of gcm_enc/dec engaged. [ 3.075063] AES CTR mode by8 optimization enabled [ 3.150192] sched_clock: Marking stable (3150116340, 0)->(4040880292, -890763952) [ 3.153523] registered taskstats version 1 [ 3.156800] Loading compiled-in X.509 certificates [ 3.159602] zswap: loaded using pool lzo/zbud [ 3.183119] Key type big_key registered [ 3.193762] Key type encrypted registered [ 3.195487] ima: No TPM chip found, activating TPM-bypass! [ 3.197182] ima: Allocated hash algorithm: sha1 [ 3.198584] ima: No architecture policies found [ 3.200167] evm: Initialising EVM extended attributes: [ 3.201403] evm: security.selinux [ 3.202298] evm: security.ima [ 3.203194] evm: security.capability [ 3.203922] evm: HMAC attrs: 0x1 [ 3.206255] rtc_cmos 00:05: setting system clock to 2026-08-27 13:45:30 UTC (1787838330) [ 3.212634] debug: unmapping init [mem 0xffffffffbe403000-0xffffffffbe5fffff] [ 3.216245] debug: unmapping init [mem 0xffffffffbd182000-0xffffffffbd458fff] [ 3.225076] Write protecting the kernel read-only data: 28672k [ 3.228075] debug: unmapping init [mem 0xffffffffbb803000-0xffffffffbb9fffff] [ 3.230894] debug: unmapping init [mem 0xffffffffbc114000-0xffffffffbc1fffff] [ 3.262146] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.268884] systemd[1]: Detected virtualization kvm. [ 3.270041] systemd[1]: Detected architecture x86-64. [ 3.271189] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.297495] systemd[1]: No hostname configured. [ 3.299243] systemd[1]: Set hostname to . [ 3.301561] random: systemd: uninitialized urandom read (16 bytes read) [ 3.304061] systemd[1]: Initializing machine ID from random generator. [ 3.422371] random: systemd: uninitialized urandom read (16 bytes read) [ 3.425245] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.433681] random: systemd: uninitialized urandom read (16 bytes read) [ 3.436183] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.440324] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.089407] device-mapper: uevent: version 1.0.3 [ 4.093250] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.917576] virtio_net virtio0 ens2: renamed from eth0 [ 4.967357] scsi host0: ata_piix [ 4.972618] scsi host1: ata_piix [ 4.977524] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.980249] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.859569] dracut-initqueue[578]: RTNETLINK answers: File exists [ 9.709318] random: crng init done [ 9.711070] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.218505] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.407919] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.675803] SELinux: Disabled at runtime. [ 11.737142] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.748402] systemd[1]: Detected virtualization kvm. [ 11.750770] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.215423] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.218236] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.228956] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.232680] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.235922] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.242764] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.255779] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK [ 12.604582] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.817838] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.531747] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.567582] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.783950] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.811441] EDAC sbridge: Ver: 1.1.2 [ 18.050528] Key type dns_resolver registered [ 18.727565] NFS: Registering the id_resolver key type [ 18.737564] Key type id_resolver registered [ 18.739337] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg317-client login: [ 63.760496] hrtimer: interrupt took 5471805 ns [ 97.428147] libcfs: loading out-of-tree module taints kernel. [ 97.600563] Key type ._llcrypt registered [ 97.612302] Key type .llcrypt registered [ 98.154843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 98.168609] alg: No test for adler32 (adler32-zlib) [ 99.827392] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 100.838888] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 102.679169] Key type lgssc registered [ 104.531806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 248.634620] Lustre: Mounted lustre-client - version 2.17.57_95_g5b39979 [ 254.390730] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 274.399221] Lustre: lustre-OST0000-osc-ffff8bf118ff8800: disconnect after 23s idle [ 275.138260] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing check_logdir /tmp/testlogs/ [ 280.721394] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing yml_node [ 285.300844] Lustre: DEBUG MARKER: Client: 2.17.57.95 [ 288.226626] Lustre: DEBUG MARKER: MDS: 2.17.57.95 [ 290.863840] Lustre: DEBUG MARKER: OSS: 2.17.57.95 [ 293.069772] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Aug 27 09:50:18 EDT 2026 [ 312.399909] Lustre: Unmounted lustre-client [ 339.455603] Key type lgssc unregistered [ 339.734738] LNet: 8450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.754908] LNetError: 8450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 339.788804] LNet: Removed LNI 192.168.203.17@tcp [ 340.574152] Key type .llcrypt unregistered [ 340.577742] Key type ._llcrypt unregistered [ 365.983616] Key type ._llcrypt registered [ 365.989709] Key type .llcrypt registered [ 366.306141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 366.318217] alg: No test for adler32 (adler32-zlib) [ 367.287825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 367.386727] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 367.434179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 372.047582] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 372.551146] LNet: 9340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.557073] LNetError: 9340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 372.581278] LNet: Removed LNI 192.168.203.17@tcp [ 373.235772] Key type .llcrypt unregistered [ 373.242034] Key type ._llcrypt unregistered [ 381.520913] Key type ._llcrypt registered [ 381.525385] Key type .llcrypt registered [ 383.143727] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:51:48 (1787838708) [ 383.505537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 383.521123] alg: No test for adler32 (adler32-zlib) [ 384.463335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.555997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.604611] LNet: 9755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.640417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.772580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 384.804470] LNet: 9762:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 391.905508] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:51:57 (1787838717) [ 392.157880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.189494] LNet: 10340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.244436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.359730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 400.060421] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:52:05 (1787838725) [ 400.516871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.557452] LNet: 10927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.584914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.710547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 407.696662] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:52:13 (1787838733) [ 407.981903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.025791] LNet: 11511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.087769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.184344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 417.567641] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:52:22 (1787838742) [ 417.744483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.813902] LNet: 12090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.833605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.987489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 418.818089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 430.449617] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 09:52:35 (1787838755) [ 430.793752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.818655] LNet: 12688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 430.854385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.939980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 431.620522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 432.135426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 440.144794] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 09:52:45 (1787838765) [ 440.236231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.289115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.420733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 447.836254] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 09:52:53 (1787838773) [ 447.997665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.032692] LNet: 13867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 448.049484] LNet: 13867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 448.116524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.220798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 448.247918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 448.810143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 448.858906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 449.272677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 449.310611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 449.811543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 449.851470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 450.282485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 450.327909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 450.774541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 450.832784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 451.346666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 451.379404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 451.840690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 451.870642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 452.368473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 452.397341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 452.817080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 452.846460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 453.076430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 453.102698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 453.348595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 453.384642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 453.983212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 454.043603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 461.173638] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 09:53:06 (1787838786) [ 461.297362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.371432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.524468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 461.561465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 468.307922] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 09:53:14 (1787838794) [ 468.611164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.658885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.803569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 468.879981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 475.670379] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 09:53:21 (1787838801) [ 475.930749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.990812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.069979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 476.128787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 483.308081] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 09:53:29 (1787838809) [ 483.545180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.576208] LNet: 16306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 483.580685] LNet: 16306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 483.625404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.787847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 483.848784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 491.791219] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 09:53:37 (1787838817) [ 491.941251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.996084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.136248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 492.178357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 498.966080] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 09:53:44 (1787838824) [ 499.083687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.133983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.237529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 499.279644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 507.108783] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 09:53:52 (1787838832) [ 507.422395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.471914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.559747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 508.195556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 515.093742] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 09:54:00 (1787838840) [ 515.236706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.295277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.617428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 515.887745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 523.265265] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 09:54:09 (1787838849) [ 523.439380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.524192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.733733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 524.074447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 530.773201] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 09:54:16 (1787838856) [ 531.104951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.174847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.692277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 532.031169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 539.416231] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 09:54:24 (1787838864) [ 539.749120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.799815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.920182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 539.998972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 547.989236] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 09:54:33 (1787838873) [ 548.123325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.163752] LNet: 21140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 548.175076] LNet: 21140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 548.220642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.377688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 548.476723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 557.564488] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 09:54:43 (1787838883) [ 557.806930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.870361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.001225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 565.641311] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 09:54:51 (1787838891) [ 565.776850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.851144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.000904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 574.044977] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 09:54:59 (1787838899) [ 574.374835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.446967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.526805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 574.982405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 581.003147] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 09:55:07 (1787838907) [ 581.235686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.269281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.346583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 581.379785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 586.731264] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 09:55:12 (1787838912) [ 586.893616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.926365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.995428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 587.029634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 593.392937] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 09:55:19 (1787838919) [ 593.483517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.528886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.615070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 593.657778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 599.975406] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 09:55:25 (1787838925) [ 600.056804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.107952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.215860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 600.249070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 606.637363] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 09:55:32 (1787838932) [ 606.834781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.883750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.938291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 607.002383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 607.022194] LNetError: 25871:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 607.068857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 607.101773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 612.682421] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 09:55:38 (1787838938) [ 612.958167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.008209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.098929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.lmZ4KLI7kJ [ 613.163537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.lmZ4KLI7kJ [ 613.257209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.lmZ4KLI7kJ [ 618.407193] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 09:55:44 (1787838944) [ 618.567613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.610293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.668682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 618.704996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 618.773350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 625.111396] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 09:55:51 (1787838951) [ 625.288844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.352651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.425655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 625.461651] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 627.191712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 628.891995] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 630.808668] Key type lgssc registered [ 632.191514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 636.617480] LNetError: 27632:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.17@tcp99 has no available nets [ 639.902632] Key type lgssc unregistered [ 640.280959] LNetError: 28743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 640.299827] LNet: Removed LNI 192.168.203.17@tcp [ 640.870803] Key type .llcrypt unregistered [ 640.879145] Key type ._llcrypt unregistered [ 647.933587] Key type ._llcrypt registered [ 647.936925] Key type .llcrypt registered [ 649.144948] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 09:56:15 (1787838975) [ 649.428392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 649.447217] alg: No test for adler32 (adler32-zlib) [ 650.320664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.411510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 652.433865] LNet: Message forwarding will be enabled in 2s [ 655.519337] LNet: Message forwarding enabled [ 655.862194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 655.885868] LNet: Message forwarding will stop in 2s to 6s [ 660.513949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 661.078347] LNet: Message forwarding disabled [ 662.391896] LNet: Message forwarding will be enabled in 2s [ 666.209111] LNet: Message forwarding enabled [ 669.869265] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 09:56:35 (1787838995) [ 669.981188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.009590] LNet: 30147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 670.830740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.904427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 677.756702] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 09:56:43 (1787839003) [ 677.872373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.919235] LNet: 30726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.951541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.088890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 678.123558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 678.169202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 678.207798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 678.237288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 678.271964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 678.304292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 678.333384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 678.369311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 678.418556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 678.452431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 678.496532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 678.532191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 678.557598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 678.584579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 678.612422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 678.637953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 678.666552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 678.697188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 678.718965] LNetError: 30777:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 684.399114] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 09:56:50 (1787839010) [ 684.631970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.664623] LNet: 31348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.686143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.753485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 690.412462] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 09:56:56 (1787839016) [ 692.048768] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg317-server [ 693.393459] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 09:56:59 (1787839019) [ 693.621518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.658508] LNet: 32278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.680608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.744505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 693.764537] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 694.517575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[18-22]@tcp [ 694.533296] LNet: 32412:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 694.577420] LNetError: 32282:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.18@tcp is being used as a gateway but routing feature is not turned on [ 694.590915] LNetError: 32282:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.18@tcp has gone from up to down [ 695.600038] LNetError: 32282:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.18@tcp is being used as a gateway but routing feature is not turned on [ 695.611086] LNetError: 32282:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 696.611849] LNetError: 32282:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.20@tcp is being used as a gateway but routing feature is not turned on [ 696.624734] LNetError: 32282:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 698.675170] LNetError: 32282:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.19@tcp is being used as a gateway but routing feature is not turned on [ 698.688939] LNetError: 32282:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 700.455960] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 09:57:06 (1787839026) [ 702.089514] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg317-server [ 702.760377] LNetError: 32282:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.18@tcp is being used as a gateway but routing feature is not turned on [ 702.773263] LNetError: 32282:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 703.495211] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 09:57:09 (1787839029) [ 703.797249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.813137] LNet: 33348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.817921] LNetError: 33348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.836253] LNet: Removed LNI 192.168.203.17@tcp [ 703.861248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.913945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 703.932906] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 704.625061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[18-23/2]@tcp [ 704.646481] LNet: 33465:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 704.655534] LNet: 33465:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 704.687760] LNetError: 33352:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.22@tcp has gone from up to down [ 704.690608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[18-23/2]@tcp [ 704.700368] LNetError: 33352:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 711.168685] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 09:57:17 (1787839037) [ 711.501217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.536838] LNetError: 34058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.551313] LNet: Removed LNI 192.168.203.17@tcp [ 711.575417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.668663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 711.711548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.740107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.809429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 711.847651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.865636] LNet: 34084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.874876] LNet: 34084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 711.900339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.930384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 711.969446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.997318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.043592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 712.117970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.147611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.180552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 712.250183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.283843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.373681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 712.447068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.532207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.588962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 712.681837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.737330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.814826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 712.900642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.948501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.005726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 713.097457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.126933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.190222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.266992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.316770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.393014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 713.501872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.542465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.578852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 720.902242] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 09:57:26 (1787839046) [ 723.650971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg317-server [ 725.789898] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 09:57:31 (1787839051) [ 727.750716] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg317-server [ 729.984675] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 09:57:35 (1787839055) [ 730.307256] LNet: 35460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.311112] LNet: 35460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 731.431722] Key type .llcrypt unregistered [ 731.437102] Key type ._llcrypt unregistered [ 739.477437] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 739.712414] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 739.720960] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 739.729987] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 742.289863] Key type ._llcrypt registered [ 742.292198] Key type .llcrypt registered [ 742.544613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 742.572594] alg: No test for adler32 (adler32-zlib) [ 743.479718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.520437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 743.540479] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 744.190595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 744.224029] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 746.208747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 746.233770] LNet: Removed LNI 10.1.2.3@tcp [ 746.360983] LNet: 36186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.366845] LNetError: 36186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.381207] LNet: Removed LNI 192.168.203.17@tcp [ 747.053464] Key type .llcrypt unregistered [ 747.060426] Key type ._llcrypt unregistered [ 758.452733] Key type ._llcrypt registered [ 758.455809] Key type .llcrypt registered [ 759.795097] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 09:58:05 (1787839085) [ 760.550683] Key type .llcrypt unregistered [ 760.555686] Key type ._llcrypt unregistered [ 766.569406] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 766.723920] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 766.727786] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 766.733223] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 769.276910] Key type ._llcrypt registered [ 769.279704] Key type .llcrypt registered [ 769.409313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 769.418580] alg: No test for adler32 (adler32-zlib) [ 770.348555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.396591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 770.416465] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 771.112410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 771.136609] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 773.556431] LNet: 37681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.568483] LNetError: 37681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 773.579733] LNet: Removed LNI 10.1.2.3@tcp [ 773.597742] LNet: Removed LNI 192.168.203.17@tcp [ 774.257688] Key type .llcrypt unregistered [ 774.261589] Key type ._llcrypt unregistered [ 785.294700] Key type ._llcrypt registered [ 785.297552] Key type .llcrypt registered [ 786.437205] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 09:58:32 (1787839112) [ 787.043491] Key type .llcrypt unregistered [ 787.046674] Key type ._llcrypt unregistered [ 792.719227] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 792.911805] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 792.918303] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 792.922788] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 795.453310] Key type ._llcrypt registered [ 795.456142] Key type .llcrypt registered [ 795.617139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 795.633953] alg: No test for adler32 (adler32-zlib) [ 796.514929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.550936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 796.576184] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 798.355925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 798.376459] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.080923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 800.105156] LNet: Removed LNI 10.1.2.3@tcp [ 800.115925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 800.158466] LNet: Removed LNI 10.1.2.31@tcp [ 800.181055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 800.210471] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.803689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 800.816119] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 801.589874] LNet: 39610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.595551] LNetError: 39610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 801.601514] LNet: Removed LNI 10.1.2.31@tcp [ 801.612142] LNet: Removed LNI 10.1.2.3@tcp [ 802.156639] Key type .llcrypt unregistered [ 802.160906] Key type ._llcrypt unregistered [ 812.071370] Key type ._llcrypt registered [ 812.074272] Key type .llcrypt registered [ 813.218579] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:58:59 (1787839139) [ 813.581499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 813.607476] alg: No test for adler32 (adler32-zlib) [ 814.541265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.618502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 814.639734] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 815.296693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.313843] LNet: 40567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.321964] LNetError: 40567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.341604] LNet: Removed LNI 192.168.203.17@tcp [ 815.365311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.413475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.17@tcp [ 815.446919] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 815.977120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.17@tcp [ 815.999521] LNet: Removed LNI 192.168.203.17@tcp [ 816.044936] LNet: 40702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.467532] Key type .llcrypt unregistered [ 816.469423] Key type ._llcrypt unregistered [ 826.810461] Key type ._llcrypt registered [ 826.813187] Key type .llcrypt registered [ 827.951311] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:59:13 (1787839153) [ 829.824347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg317-server [ 831.510985] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:59:17 (1787839157) [ 832.002441] Key type .llcrypt unregistered [ 832.005916] Key type ._llcrypt unregistered [ 837.744305] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 837.933331] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 837.943729] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 837.956325] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 840.480681] Key type ._llcrypt registered [ 840.483110] Key type .llcrypt registered [ 840.653290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 840.668381] alg: No test for adler32 (adler32-zlib) [ 841.652864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.734120] LNet: Added LNI 192.168.203.17@tcp1 [8/256/0/180] [ 842.446568] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 844.538872] LNet: 42542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.545666] LNetError: 42542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.556469] LNet: Removed LNI 192.168.203.17@tcp1 [ 844.567743] LNet: Removed LNI 10.1.2.3@tcp2 [ 845.169186] Key type .llcrypt unregistered [ 845.175648] Key type ._llcrypt unregistered [ 854.777250] Key type ._llcrypt registered [ 854.780338] Key type .llcrypt registered [ 855.793595] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:59:41 (1787839181) [ 856.295474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 856.314733] alg: No test for adler32 (adler32-zlib) [ 857.271946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.311946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 857.333751] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 858.051254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.072426] LNet: 43515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.082785] LNetError: 43515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.096301] LNet: Removed LNI 192.168.203.17@tcp [ 858.118296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.187462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 858.222165] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 864.946569] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:59:50 (1787839190) [ 865.063060] LNet: 44099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.071818] LNetError: 44099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.093482] LNet: Removed LNI 192.168.203.17@tcp [ 865.801270] Key type .llcrypt unregistered [ 865.804279] Key type ._llcrypt unregistered [ 871.869251] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 872.054692] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 872.066923] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 872.072170] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 874.636933] Key type ._llcrypt registered [ 874.641217] Key type .llcrypt registered [ 874.904298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 874.929646] alg: No test for adler32 (adler32-zlib) [ 875.843857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.905548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 875.935844] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 876.615032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 876.632339] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 878.847594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 878.880940] LNet: 44848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.888828] LNetError: 44848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 878.908624] LNet: Removed LNI 10.1.2.3@tcp [ 878.934692] LNet: Removed LNI 192.168.203.17@tcp [ 878.972674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.032731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 879.078742] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 879.085181] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 879.643334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.697524] LNet: 44875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.707086] LNetError: 44875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 879.716437] LNetError: 44875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 879.720682] LNet: Removed LNI 10.1.2.3@tcp [ 879.736228] LNet: Removed LNI 192.168.203.17@tcp [ 879.753438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.829756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 879.865699] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 879.875150] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 880.432639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 880.456637] LNet: Added LNI 192.168.203.17@tcp2 [8/256/0/180] [ 881.541569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.557441] LNet: 45037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.562599] LNetError: 45037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 881.567220] LNetError: 45037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 881.576415] LNet: Removed LNI 10.1.2.3@tcp [ 881.581441] LNet: Removed LNI 192.168.203.17@tcp [ 881.591730] LNet: Removed LNI 192.168.203.17@tcp2 [ 881.618143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.691164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 881.730291] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 881.739140] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 881.746789] LNet: Added LNI 192.168.203.17@tcp2 [8/256/0/180] [ 888.751551] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 10:00:14 (1787839214) [ 888.848786] LNet: 45630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.856651] LNetError: 45630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 888.861347] LNetError: 45630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 888.869089] LNet: Removed LNI 10.1.2.3@tcp [ 888.872756] LNet: Removed LNI 192.168.203.17@tcp [ 888.887618] LNet: Removed LNI 192.168.203.17@tcp2 [ 889.398776] Key type .llcrypt unregistered [ 889.403045] Key type ._llcrypt unregistered [ 895.670818] Key type ._llcrypt registered [ 895.673434] Key type .llcrypt registered [ 895.836123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 895.850268] alg: No test for adler32 (adler32-zlib) [ 896.778834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.853777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 896.880989] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 897.919476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.949356] LNet: 46033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.955744] LNetError: 46033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.970433] LNet: Removed LNI 192.168.203.17@tcp [ 897.988687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.083341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 898.110798] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 898.701226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.731675] LNet: 46063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.740052] LNetError: 46063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.760514] LNet: Removed LNI 192.168.203.17@tcp [ 898.781587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.835084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 898.859844] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 899.354490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.391462] LNet: Removed LNI 192.168.203.17@tcp [ 899.443980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.582622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 899.624295] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 900.233779] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 900.464938] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 900.470624] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 900.476850] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 902.960871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 902.981642] LNet: 46171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.988436] LNet: 46171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 902.995788] LNetError: 46171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.000826] LNetError: 46171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 903.021118] LNet: Removed LNI 192.168.203.17@tcp [ 903.048364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.097724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 903.114053] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 909.169694] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 10:00:35 (1787839235) [ 909.244665] LNet: 46760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.251215] LNetError: 46760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.267265] LNet: Removed LNI 192.168.203.17@tcp [ 910.032497] Key type .llcrypt unregistered [ 910.035618] Key type ._llcrypt unregistered [ 916.729266] Key type ._llcrypt registered [ 916.750168] Key type .llcrypt registered [ 917.011292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 917.038616] alg: No test for adler32 (adler32-zlib) [ 918.036376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.104789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 918.147250] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 919.008241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.024925] LNet: 47172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.040033] LNetError: 47172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.055263] LNet: Removed LNI 192.168.203.17@tcp [ 919.086216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.158735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 919.201213] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 924.662375] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 10:00:50 (1787839250) [ 924.784235] LNet: 47759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.792765] LNetError: 47759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.808034] LNet: Removed LNI 192.168.203.17@tcp [ 925.427525] Key type .llcrypt unregistered [ 925.433102] Key type ._llcrypt unregistered [ 930.989787] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 931.076943] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 931.083182] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 931.087829] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 933.875413] Key type ._llcrypt registered [ 933.881637] Key type .llcrypt registered [ 934.140290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 934.164216] alg: No test for adler32 (adler32-zlib) [ 935.136504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.228491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 935.269531] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 936.220772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.239496] LNet: 48232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.246141] LNetError: 48232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.272093] LNet: Removed LNI 192.168.203.17@tcp [ 936.311317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.403991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 936.435883] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 942.622507] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 10:01:08 (1787839268) [ 942.868745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 942.888909] LNet: 48830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.896194] LNetError: 48830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.913784] LNet: Removed LNI 192.168.203.17@tcp [ 942.946531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.080267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 948.932342] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 10:01:14 (1787839274) [ 949.069803] LNet: 49407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.865630] Key type .llcrypt unregistered [ 949.870525] Key type ._llcrypt unregistered [ 957.096191] Key type ._llcrypt registered [ 957.102141] Key type .llcrypt registered [ 957.359594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 957.384876] alg: No test for adler32 (adler32-zlib) [ 958.336727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.419813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 958.464594] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 959.501974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 959.524747] LNet: 49812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.536246] LNetError: 49812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.552473] LNet: Removed LNI 192.168.203.17@tcp [ 959.572107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.645581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 959.679687] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 966.368517] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 10:01:32 (1787839292) [ 966.514854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.544482] LNet: 50399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.559599] LNetError: 50399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.577590] LNet: Removed LNI 192.168.203.17@tcp [ 966.624086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.713229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 973.658960] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 10:01:39 (1787839299) [ 975.187439] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 976.693828] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 10:01:42 (1787839302) [ 978.287329] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 980.442120] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 10:01:46 (1787839306) [ 982.173153] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 983.846242] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 10:01:49 (1787839309) [ 985.447224] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 986.864900] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 10:01:53 (1787839313) [ 988.332143] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 990.314401] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 10:01:56 (1787839316) [ 990.500307] LNet: 52458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.222652] Key type .llcrypt unregistered [ 991.226599] Key type ._llcrypt unregistered [ 998.517359] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 998.610702] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 998.616875] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 998.621551] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1001.253899] Key type ._llcrypt registered [ 1001.258399] Key type .llcrypt registered [ 1001.442379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1001.460678] alg: No test for adler32 (adler32-zlib) [ 1002.409659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.533787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1002.566818] LNet: Added LNI 192.168.203.17@tcp [8/512/0/180] [ 1002.573981] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1002.579649] LNet: 52785:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1002.703345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1002.725144] LNet: 52804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.732617] LNetError: 52804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1002.741123] LNet: Removed LNI 10.1.2.3@tcp [ 1002.744033] LNetError: 52804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.755602] LNet: Removed LNI 192.168.203.17@tcp [ 1002.777285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.820089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1002.843360] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1002.887090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1002.912270] LNet: 52828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.918331] LNetError: 52828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.934894] LNet: Removed LNI 192.168.203.17@tcp [ 1002.957489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.022498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1003.039640] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1003.043949] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1003.045750] LNet: 52836:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1003.112842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.130735] LNet: 52854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.136754] LNetError: 52854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1003.142101] LNet: Removed LNI 10.1.2.3@tcp [ 1003.144544] LNetError: 52854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.155823] LNet: Removed LNI 192.168.203.17@tcp [ 1003.172687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.220866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1003.247154] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1003.255689] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1003.260604] LNet: 52862:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1003.413168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.444627] LNet: 52888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.449279] LNetError: 52888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1003.455318] LNet: Removed LNI 10.1.2.3@tcp [ 1003.474908] LNetError: 52888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.497875] LNet: Removed LNI 192.168.203.17@tcp [ 1003.532052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.608833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1003.646632] LNet: Added LNI 192.168.203.17@tcp [8/256/0/182] [ 1003.658131] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1003.767672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.790108] LNet: 52916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.794993] LNetError: 52916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1003.799107] LNet: Removed LNI 10.1.2.3@tcp [ 1003.801116] LNetError: 52916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.813976] LNet: Removed LNI 192.168.203.17@tcp [ 1003.827076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.891796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1003.919971] LNet: Added LNI 192.168.203.17@tcp [8/256/0/182] [ 1003.925616] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1003.928191] LNet: 52924:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1004.020988] LNet: Removed LNI 10.1.2.3@tcp [ 1004.038889] LNet: Removed LNI 192.168.203.17@tcp [ 1004.666808] Key type .llcrypt unregistered [ 1004.672504] Key type ._llcrypt unregistered [ 1014.827415] Key type ._llcrypt registered [ 1014.832290] Key type .llcrypt registered [ 1015.994865] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 10:02:21 (1787839341) [ 1017.353555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1019.000670] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:02:25 (1787839345) [ 1019.494876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1019.514809] alg: No test for adler32 (adler32-zlib) [ 1020.399452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.443461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1020.464989] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1021.127232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.17 [ 1021.153271] LNet: 54193:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1021.160793] LNet: Removed LNI 192.168.203.17@tcp [ 1027.399246] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:02:33 (1787839353) [ 1027.595107] LNet: 54765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.606949] LNetError: 54765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.625100] LNet: Removed LNI 192.168.203.17@tcp [ 1028.197768] Key type .llcrypt unregistered [ 1028.201669] Key type ._llcrypt unregistered [ 1033.552434] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1033.774711] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1036.436546] Key type ._llcrypt registered [ 1036.440993] Key type .llcrypt registered [ 1036.626739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1036.640652] alg: No test for adler32 (adler32-zlib) [ 1037.804727] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1043.393251] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:02:49 (1787839369) [ 1043.728460] LNet: 55682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.737392] LNetError: 55682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1043.764787] LNet: Removed LNI 10.1.2.3@tcp [ 1044.429568] Key type .llcrypt unregistered [ 1044.437115] Key type ._llcrypt unregistered [ 1050.358886] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1050.584058] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1050.588788] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1050.593325] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1053.202447] Key type ._llcrypt registered [ 1053.204172] Key type .llcrypt registered [ 1053.383753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1053.405223] alg: No test for adler32 (adler32-zlib) [ 1054.375615] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1061.012973] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:03:06 (1787839386) [ 1062.583502] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1064.330878] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:03:10 (1787839390) [ 1064.533845] LNet: 56872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.547215] LNetError: 56872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1064.581875] LNet: Removed LNI 10.1.2.3@tcp [ 1065.248381] Key type .llcrypt unregistered [ 1065.250102] Key type ._llcrypt unregistered [ 1072.024641] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1072.202915] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1072.206871] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1072.211186] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1074.896466] Key type ._llcrypt registered [ 1074.898541] Key type .llcrypt registered [ 1075.027962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1075.047934] alg: No test for adler32 (adler32-zlib) [ 1075.953290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1076.043151] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1083.982475] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:03:30 (1787839410) [ 1084.297559] LNet: 58107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.303093] LNetError: 58107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1084.316940] LNet: Removed LNI 10.1.2.3@tcp [ 1084.872872] Key type .llcrypt unregistered [ 1084.875863] Key type ._llcrypt unregistered [ 1090.816941] Key type ._llcrypt registered [ 1090.819747] Key type .llcrypt registered [ 1090.992274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1091.010073] alg: No test for adler32 (adler32-zlib) [ 1091.927639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1091.976266] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1098.982651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.117@tcp [ 1099.532767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_interrupt -r 1 [ 1099.562931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_interrupt -r 1 [ 1099.593518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1099.611569] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1099.814881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1099.853856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1099.892833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1099.925228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1099.955660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1099.989367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.025351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.058863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.082325] LNet: 58549:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.111294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.144158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.167606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1100.192140] LNet: 1 local NIs in recovery (showing 1): 192.168.203.17@tcp [ 1100.197397] LNet: There was an unexpected network error while writing to 192.168.203.17: rc = -22 [ 1100.206436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1100.239198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1100.271369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1100.309631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1100.891779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_dropped -r 1 [ 1100.949068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_dropped -r 1 [ 1100.988537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1101.239693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1101.292141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1101.358167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1101.379614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1101.405966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1101.446694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1101.514551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1101.603659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1101.657402] LNet: 58658:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1101.695610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1101.767886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1101.814916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1101.863663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1101.948211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1101.996285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1102.062624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1102.622731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_aborted -r 1 [ 1102.650728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_aborted -r 1 [ 1102.683509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1102.698906] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1102.702454] LNet: Skipped 1 previous similar message [ 1102.838298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1102.873583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1102.904860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1102.941103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.962144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1102.985271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1103.014537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1103.067127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1103.108367] LNet: 58767:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1103.170801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1103.197271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1103.226076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1103.266640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1103.303573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1103.346263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1103.396852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1103.962408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_no_route -r 1 [ 1104.001342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_no_route -r 1 [ 1104.038981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1104.303902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.350537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1104.396568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.443905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.496576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.550671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.586382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.612546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.659608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.691778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.733263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.753902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.797559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.829902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.857772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1105.231554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_timeout -r 1 [ 1105.249451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_timeout -r 1 [ 1105.271715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1105.292412] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1105.299364] LNet: Skipped 2 previous similar messages [ 1105.503030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.530202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1105.545140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1105.585542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1105.624907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1105.673399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1105.750544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.812676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1105.896645] LNet: 58986:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1105.899891] LNet: 58986:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1105.950890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.011348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.077918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.132150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.157022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.242335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.309355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1106.707448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_error -r 1 [ 1106.774270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_error -r 1 [ 1106.819292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1106.956572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.977800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1107.001339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.026621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.049317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.074307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.103163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.160565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.187671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.239833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.273130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.293668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.312271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.344761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.384224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1107.761929] LNet: 59140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.769915] LNetError: 59140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1108.846854] LNet: Removed LNI 192.168.203.17@tcp [ 1109.366594] Key type .llcrypt unregistered [ 1109.370698] Key type ._llcrypt unregistered [ 1136.603276] Key type ._llcrypt registered [ 1136.606104] Key type .llcrypt registered [ 1138.073609] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:04:23 (1787839463) [ 1139.045587] Key type .llcrypt unregistered [ 1139.047408] Key type ._llcrypt unregistered [ 1144.779394] Key type ._llcrypt registered [ 1144.785258] Key type .llcrypt registered [ 1145.265664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1145.307044] alg: No test for adler32 (adler32-zlib) [ 1146.325896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1146.351186] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1155.703401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.117@tcp [ 1166.934446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1166.993876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1167.025937] LNet: Added LNI 192.168.203.17@tcp1 [8/256/0/180] [ 1168.148835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_interrupt -r 1 [ 1168.178416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp1 -m GET -e local_interrupt -r 1 [ 1168.212958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_interrupt -r 1 [ 1168.243201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -m GET -e local_interrupt -r 1 [ 1168.296738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp -m GET -e local_interrupt -r 1 [ 1168.326478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp1 -m GET -e local_interrupt -r 1 [ 1168.364590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -m GET -e local_interrupt -r 1 [ 1168.405994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -m GET -e local_interrupt -r 1 [ 1168.436760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1168.453288] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1168.457981] LNet: 1 local NIs in recovery (showing 1): 192.168.203.17@tcp [ 1168.642720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1168.663977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1168.681599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1168.699806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1168.727990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1168.749326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1168.768784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1168.792766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.808661] LNet: 60615:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.824267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.862426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1168.903388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.948199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.999420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.030177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.063366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1169.520639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_dropped -r 1 [ 1169.555486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp1 -m GET -e local_dropped -r 1 [ 1169.590954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_dropped -r 1 [ 1169.627513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -m GET -e local_dropped -r 1 [ 1169.659636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp -m GET -e local_dropped -r 1 [ 1169.689996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp1 -m GET -e local_dropped -r 1 [ 1169.727383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -m GET -e local_dropped -r 1 [ 1169.757527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -m GET -e local_dropped -r 1 [ 1169.795775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1169.813592] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1169.822744] LNet: Skipped 2 previous similar messages [ 1170.029742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.056840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1170.080780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1170.121745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.144572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.171419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.210984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.233209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.244798] LNet: 60736:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1170.257750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.289932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.316456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.354547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.390920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.429947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.466247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1171.035370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_aborted -r 1 [ 1171.064355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp1 -m GET -e local_aborted -r 1 [ 1171.092797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_aborted -r 1 [ 1171.118518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -m GET -e local_aborted -r 1 [ 1171.146391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp -m GET -e local_aborted -r 1 [ 1171.176414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp1 -m GET -e local_aborted -r 1 [ 1171.201625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -m GET -e local_aborted -r 1 [ 1171.228944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -m GET -e local_aborted -r 1 [ 1171.259932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1171.275176] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1171.279615] LNet: Skipped 2 previous similar messages [ 1171.427386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.457744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1171.493689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.523921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.569921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.607582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.637273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.662076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.682385] LNet: 60858:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.704907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.742637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.781621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.819404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.865313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.901144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.948844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.423338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_no_route -r 1 [ 1172.469487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp1 -m GET -e local_no_route -r 1 [ 1172.521103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_no_route -r 1 [ 1172.576948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -m GET -e local_no_route -r 1 [ 1172.616534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp -m GET -e local_no_route -r 1 [ 1172.643836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp1 -m GET -e local_no_route -r 1 [ 1172.667167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -m GET -e local_no_route -r 1 [ 1172.700699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -m GET -e local_no_route -r 1 [ 1172.733704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1172.936241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.957465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1172.993429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.020263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.044176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.077878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.106360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.134480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.167385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.192644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.215554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.250714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.279807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.304249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.333566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1173.748485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_timeout -r 1 [ 1173.798366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp1 -m GET -e local_timeout -r 1 [ 1173.840471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_timeout -r 1 [ 1173.880378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -m GET -e local_timeout -r 1 [ 1173.922868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp -m GET -e local_timeout -r 1 [ 1173.958988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp1 -m GET -e local_timeout -r 1 [ 1173.994798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -m GET -e local_timeout -r 1 [ 1174.023348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -m GET -e local_timeout -r 1 [ 1174.058752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1174.089947] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1174.096791] LNet: Skipped 5 previous similar messages [ 1174.310441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.345904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.377080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.398389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.431713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.460289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.488654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.520102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.550658] LNet: 61100:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1174.564428] LNet: 61100:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1174.591360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.639288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.671687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.702207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.738194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.784612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.806680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1175.267467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_error -r 1 [ 1175.309581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp1 -m GET -e local_error -r 1 [ 1175.346630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_error -r 1 [ 1175.387937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -m GET -e local_error -r 1 [ 1175.422777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp -m GET -e local_error -r 1 [ 1175.470831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp1 -m GET -e local_error -r 1 [ 1175.509564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -m GET -e local_error -r 1 [ 1175.545609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -m GET -e local_error -r 1 [ 1175.597554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1175.823433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1175.857598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1175.892315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.929899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.974071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.005861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.038906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.075423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.106260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1176.154977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.197576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.244916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.284934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.317434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.360955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1180.743917] LNet: 61302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1180.752141] LNetError: 61302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1180.760712] LNet: Removed LNI 192.168.203.17@tcp [ 1181.806657] LNet: Removed LNI 192.168.203.17@tcp1 [ 1182.119675] Key type .llcrypt unregistered [ 1182.122881] Key type ._llcrypt unregistered [ 1195.502529] Key type ._llcrypt registered [ 1195.505495] Key type .llcrypt registered [ 1196.667317] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:05:22 (1787839522) [ 1197.305347] Key type .llcrypt unregistered [ 1197.308191] Key type ._llcrypt unregistered [ 1204.845898] Key type ._llcrypt registered [ 1204.848539] Key type .llcrypt registered [ 1205.058498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1205.076204] alg: No test for adler32 (adler32-zlib) [ 1205.984406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1206.006568] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1215.009970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.117@tcp [ 1215.753020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e remote_dropped -r 1 [ 1215.790647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e remote_dropped -r 1 [ 1215.845595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1215.878204] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1216.170870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1216.246251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1216.301951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1216.334158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1216.364197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1216.393777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1216.419693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1216.444549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1216.466660] LNet: 62555:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1216.493536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1216.516237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1216.548121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1216.573251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1216.599497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1216.642667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1216.694039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1217.175930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e remote_error -r 1 [ 1217.234619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e remote_error -r 1 [ 1217.295464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1217.319042] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1217.582431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1217.618589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1217.674211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1217.698963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1217.728753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1217.763817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1217.790162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1217.832741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1217.863977] LNet: 62665:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1217.911563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1217.959614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1218.000375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1218.035155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1218.071765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1218.097853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1218.130438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1218.758874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e remote_timeout -r 1 [ 1218.813576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e remote_timeout -r 1 [ 1218.900209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1218.944526] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1219.210181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1219.241091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1219.276420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1219.347457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1219.395043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1219.473538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1219.516080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1219.597273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1219.638242] LNet: 62774:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1219.664365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1219.703626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1219.743334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1219.798356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1219.857236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1219.909463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1219.955789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1220.304463] LNet: 62819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1220.310715] LNetError: 62819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1220.337366] LNet: Removed LNI 192.168.203.17@tcp [ 1220.814715] Key type .llcrypt unregistered [ 1220.816426] Key type ._llcrypt unregistered [ 1238.419419] Key type ._llcrypt registered [ 1238.422071] Key type .llcrypt registered [ 1239.739446] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:06:05 (1787839565) [ 1240.520645] Key type .llcrypt unregistered [ 1240.524809] Key type ._llcrypt unregistered [ 1247.191695] Key type ._llcrypt registered [ 1247.194100] Key type .llcrypt registered [ 1247.422551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1247.450949] alg: No test for adler32 (adler32-zlib) [ 1248.486256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1248.535607] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1256.474711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.117@tcp [ 1266.007759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1266.099624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1266.128571] LNet: Added LNI 192.168.203.17@tcp1 [8/256/0/180] [ 1268.007780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e remote_dropped -r 1 [ 1268.089394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp1 -m GET -e remote_dropped -r 1 [ 1268.168958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e remote_dropped -r 1 [ 1268.234958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -m GET -e remote_dropped -r 1 [ 1268.287231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp -m GET -e remote_dropped -r 1 [ 1268.356787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp1 -m GET -e remote_dropped -r 1 [ 1268.424647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -m GET -e remote_dropped -r 1 [ 1268.479211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -m GET -e remote_dropped -r 1 [ 1268.580453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1268.608917] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1268.616768] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.117@tcp [ 1268.968977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1269.016892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1269.066784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1269.103909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1269.150073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1269.222358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1269.287269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1269.359183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1269.402171] LNet: 64289:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1269.443697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1269.487945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1269.527352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1269.563944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1269.585171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1269.615097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1269.638534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1269.664244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1270.155445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e remote_error -r 1 [ 1270.187143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp1 -m GET -e remote_error -r 1 [ 1270.228408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e remote_error -r 1 [ 1270.265165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -m GET -e remote_error -r 1 [ 1270.316641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp -m GET -e remote_error -r 1 [ 1270.355162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp1 -m GET -e remote_error -r 1 [ 1270.395771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -m GET -e remote_error -r 1 [ 1270.433887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -m GET -e remote_error -r 1 [ 1270.483940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1270.525699] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1270.529982] LNet: Skipped 2 previous similar messages [ 1270.849221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1270.884800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1270.926272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1270.973962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1271.037429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1271.062460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1271.094211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1271.121772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1271.135640] LNet: 64412:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1271.156505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1271.199560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1271.224550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1271.284748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1271.321449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1271.355576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1271.434180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1271.496977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1271.893945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e remote_timeout -r 1 [ 1271.933959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp1 -m GET -e remote_timeout -r 1 [ 1271.974980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e remote_timeout -r 1 [ 1272.020523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -m GET -e remote_timeout -r 1 [ 1272.056648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp -m GET -e remote_timeout -r 1 [ 1272.104805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp1 -m GET -e remote_timeout -r 1 [ 1272.135547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -m GET -e remote_timeout -r 1 [ 1272.178678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -m GET -e remote_timeout -r 1 [ 1272.247274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.117@tcp [ 1272.295829] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1272.470934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1272.502428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1272.535712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1272.563236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1272.602716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1272.639978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1272.655141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1272.673910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1272.686127] LNet: 64535:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1272.697591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1272.732357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1272.757348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1272.788839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1272.848426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1272.890549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1272.940135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1272.982679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.258946] LNet: 64618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.266285] LNetError: 64618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1278.270602] LNet: Removed LNI 192.168.203.17@tcp [ 1278.279355] LNet: Removed LNI 192.168.203.17@tcp1 [ 1278.685288] Key type .llcrypt unregistered [ 1278.687659] Key type ._llcrypt unregistered [ 1293.313984] Key type ._llcrypt registered [ 1293.318240] Key type .llcrypt registered [ 1294.614171] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:07:00 (1787839620) [ 1296.220773] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1297.986911] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:07:03 (1787839623) [ 1298.594501] Key type .llcrypt unregistered [ 1298.599233] Key type ._llcrypt unregistered [ 1304.575855] Key type ._llcrypt registered [ 1304.578602] Key type .llcrypt registered [ 1304.717070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1304.728358] alg: No test for adler32 (adler32-zlib) [ 1305.692547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1305.721286] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1313.140877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.117@tcp [ 1313.820773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e network_timeout -r 1 [ 1313.863818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e network_timeout -r 1 [ 1313.922652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.117@tcp [ 1313.941473] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1314.220989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1314.264967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1314.310795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1314.359759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1314.420453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1314.478480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1314.547434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1314.595358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1314.618270] LNet: 66164:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1314.641778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1314.686371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1314.723735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1314.774972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1314.809327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1314.839744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1315.076730] LNet: 66207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1315.081979] LNetError: 66207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1315.098651] LNet: Removed LNI 192.168.203.17@tcp [ 1315.420622] Key type .llcrypt unregistered [ 1315.425908] Key type ._llcrypt unregistered [ 1332.102151] Key type ._llcrypt registered [ 1332.106992] Key type .llcrypt registered [ 1332.324745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1332.343489] alg: No test for adler32 (adler32-zlib) [ 1333.242756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1333.264142] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1340.372701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.117@tcp [ 1349.253416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1349.301634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1349.335448] LNet: Added LNI 192.168.203.17@tcp1 [8/256/0/180] [ 1350.801790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e network_timeout -r 1 [ 1350.838849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp1 -m GET -e network_timeout -r 1 [ 1350.883094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e network_timeout -r 1 [ 1350.910786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -m GET -e network_timeout -r 1 [ 1350.975162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp -m GET -e network_timeout -r 1 [ 1351.030159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp1 -m GET -e network_timeout -r 1 [ 1351.076275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -m GET -e network_timeout -r 1 [ 1351.127625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -m GET -e network_timeout -r 1 [ 1351.182356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.117@tcp [ 1351.206320] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1351.420340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1351.466525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1351.488802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1351.529317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1351.573124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1351.604857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1351.635774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1351.661975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1351.686943] LNet: 67090:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1351.705853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1351.713104] LNet: 1 local NIs in recovery (showing 1): 192.168.203.17@tcp [ 1351.715818] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.117@tcp [ 1351.749340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1351.804992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1351.844882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1351.869389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1351.893217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1356.252694] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1356.610863] LNet: 67167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1356.617619] LNetError: 67167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1356.623915] LNet: Removed LNI 192.168.203.17@tcp [ 1356.633788] LNet: Removed LNI 192.168.203.17@tcp1 [ 1356.974485] Key type .llcrypt unregistered [ 1356.976774] Key type ._llcrypt unregistered [ 1372.048586] Key type ._llcrypt registered [ 1372.050661] Key type .llcrypt registered [ 1373.077466] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:08:19 (1787839699) [ 1373.568412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1373.588911] alg: No test for adler32 (adler32-zlib) [ 1374.540378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.598939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1374.619714] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1375.187632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1375.203451] LNet: Added LNI 192.168.203.17@tcp1 [8/256/0/180] [ 1375.911206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 1375.986578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1376.023617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1376.049159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1376.089168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1376.127521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1376.168168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.17@tcp [ 1376.198520] LNetError: 68268:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.17@tcp to recovery [ 1376.607298] LNet: 1 local NIs in recovery (showing 1): 192.168.203.17@tcp [ 1376.608375] LNet: There was an unexpected network error while writing to 192.168.203.17: rc = -22 [ 1378.656073] LNet: There was an unexpected network error while writing to 192.168.203.17: rc = -22 [ 1379.687970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1382.752677] LNet: There was an unexpected network error while writing to 192.168.203.17: rc = -22 [ 1383.252455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1383.285268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1383.331941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1383.343720] LNet: 68375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.350530] LNetError: 68375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.357278] LNet: Removed LNI 192.168.203.17@tcp [ 1383.373828] LNet: Removed LNI 192.168.203.17@tcp1 [ 1383.414098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1383.503709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1383.563087] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1384.270406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1384.293821] LNet: Added LNI 192.168.203.17@tcp1 [8/256/0/180] [ 1385.263283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 1385.323647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1385.415949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1385.493546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1385.553485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1385.600332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1385.644526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1385.680895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.17@tcp [ 1385.705699] LNetError: 68647:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.17@tcp to recovery [ 1386.527381] LNet: There was an unexpected network error while writing to 192.168.203.17: rc = -22 [ 1389.211445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1392.673769] LNet: There was an unexpected network error while writing to 192.168.203.17: rc = -22 [ 1392.682924] LNet: Skipped 1 previous similar message [ 1397.254466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1397.315578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1397.377627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1404.761493] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:08:50 (1787839730) [ 1404.878733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1404.919486] LNet: 69357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1404.944745] LNetError: 69357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1404.959206] LNetError: 69357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1404.974483] LNet: Removed LNI 192.168.203.17@tcp [ 1404.988224] LNet: Removed LNI 192.168.203.17@tcp1 [ 1405.030775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1405.139306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1405.188988] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1406.031701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1406.056743] LNet: Added LNI 192.168.203.17@tcp1 [8/256/0/180] [ 1407.024036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 1407.115792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1407.145774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1407.196498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1407.266374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1407.318525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1407.380406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.17@tcp --health 0 [ 1407.536388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1408.159921] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.17@tcp [ 1408.170874] LNet: There was an unexpected network error while writing to 192.168.203.17: rc = -22 [ 1408.193317] LNet: Skipped 1 previous similar message [ 1422.957567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1423.116148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1423.144495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1423.188443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.17@tcp --health 500 [ 1423.348042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1430.000264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1430.081269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1430.122466] LNet: 69856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1430.131164] LNetError: 69856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1430.144074] LNetError: 69856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1430.150937] LNet: Removed LNI 192.168.203.17@tcp [ 1430.163720] LNet: Removed LNI 192.168.203.17@tcp1 [ 1430.185946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1430.262197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1430.294214] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1431.095868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1431.133792] LNet: Added LNI 192.168.203.17@tcp1 [8/256/0/180] [ 1432.350307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 1432.423260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1432.552549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1432.605512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1432.635107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1432.672444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1432.709656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1432.739644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.17@tcp --health 0 [ 1433.311345] LNet: There was an unexpected network error while writing to 192.168.203.17: rc = -22 [ 1433.317773] LNet: Skipped 3 previous similar messages [ 1434.047275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1444.375901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1444.409286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1444.440327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1451.228565] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:09:37 (1787839777) [ 1451.735769] LNet: 70903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.762339] LNetError: 70903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.769249] LNetError: 70903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1451.774783] LNet: Removed LNI 192.168.203.17@tcp [ 1451.782848] LNet: Removed LNI 192.168.203.17@tcp1 [ 1452.510223] Key type .llcrypt unregistered [ 1452.517136] Key type ._llcrypt unregistered [ 1457.756303] Key type ._llcrypt registered [ 1457.758495] Key type .llcrypt registered [ 1457.940428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1457.954825] alg: No test for adler32 (adler32-zlib) [ 1458.869588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1458.897095] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1465.886103] Lustre: DEBUG MARKER: Initial discovery [ 1466.180380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.117@tcp [ 1468.054808] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1468.221379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1468.259117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.117@tcp [ 1468.289057] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1468.312325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1468.358869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.117@tcp -r 1 -m GET -l 3 [ 1468.413208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.117@tcp -r 1 -m GET -e local_error [ 1468.444846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.117@tcp -r 1 -m PUT -l 6 [ 1469.152327] LNet: 1 local NIs in recovery (showing 1): 192.168.203.17@tcp [ 1469.564359] Lustre: DEBUG MARKER: Force oleg317-client.virtnet to discover 192.168.203.117@tcp (in background) [ 1469.866838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.117@tcp [ 1471.007861] Lustre: DEBUG MARKER: Force oleg317-server to discover 192.168.203.17@tcp [ 1472.934242] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1473.388134] Lustre: DEBUG MARKER: Wait for 71670 [ 1474.603587] Lustre: DEBUG MARKER: Finished wait on 71670 [ 1474.644497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1474.675183] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1474.677534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1474.973529] LNet: 72122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.979646] LNetError: 72122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1476.010882] LNet: Removed LNI 192.168.203.17@tcp [ 1476.422666] Key type .llcrypt unregistered [ 1476.428523] Key type ._llcrypt unregistered [ 1490.954849] Key type ._llcrypt registered [ 1490.957742] Key type .llcrypt registered [ 1492.248352] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:10:18 (1787839818) [ 1492.834555] Key type .llcrypt unregistered [ 1492.837645] Key type ._llcrypt unregistered [ 1498.788516] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1498.888817] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1498.896147] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1498.900781] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1501.403450] Key type ._llcrypt registered [ 1501.405207] Key type .llcrypt registered [ 1501.558237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1501.569803] alg: No test for adler32 (adler32-zlib) [ 1502.499695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1502.555704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1502.585063] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1503.212837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1503.246129] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1510.265428] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 10:10:36 (1787839836) [ 1510.411473] LNet: 74199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1510.421360] LNetError: 74199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1510.432562] LNet: Removed LNI 10.1.2.3@tcp [ 1510.461738] LNet: Removed LNI 192.168.203.17@tcp [ 1511.143461] Key type .llcrypt unregistered [ 1511.148977] Key type ._llcrypt unregistered [ 1518.138484] Key type ._llcrypt registered [ 1518.141933] Key type .llcrypt registered [ 1518.375210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1518.389930] alg: No test for adler32 (adler32-zlib) [ 1519.290695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.351836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1519.377928] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1520.096329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1520.118915] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1520.970145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.17@tcp [ 1522.015245] LNetError: Unexpected error -22 connecting to 192.168.203.17@tcp at host 192.168.203.17:988 [ 1526.540569] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 10:10:52 (1787839852) [ 1526.796950] LNet: 75338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.802676] LNetError: 75338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1526.811370] LNet: Removed LNI 10.1.2.3@tcp [ 1526.818572] LNet: Removed LNI 192.168.203.17@tcp [ 1527.418523] Key type .llcrypt unregistered [ 1527.421193] Key type ._llcrypt unregistered [ 1532.596501] Key type ._llcrypt registered [ 1532.601520] Key type .llcrypt registered [ 1532.751490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1532.781561] alg: No test for adler32 (adler32-zlib) [ 1533.725806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1533.800541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1533.828911] LNet: Added LNI 192.168.203.17@tcp1 [8/256/0/180] [ 1534.589443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1534.606716] LNet: Added LNI 192.168.203.17@tcp2 [8/256/0/180] [ 1535.352457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.17@tcp1 --nid 192.168.203.17@tcp2 [ 1538.427443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.17@tcp1 192.168.203.17@tcp2 [ 1538.450801] LNetError: 76177:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.17@tcp1 to 192.168.203.17@tcp2 - Route table may be misconfigured [ 1538.478211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.17@tcp2 192.168.203.17@tcp1 [ 1544.257239] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 10:11:10 (1787839870) [ 1544.521077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1544.545526] LNet: 76750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.552305] LNetError: 76750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1545.567263] LNet: Removed LNI 192.168.203.17@tcp1 [ 1545.570372] LNetError: 76750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.616839] LNet: Removed LNI 192.168.203.17@tcp2 [ 1546.644764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.691674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1546.717920] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1547.340575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1547.358940] LNet: Added LNI 192.168.203.17@tcp1 [8/256/0/180] [ 1548.073798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 1548.107355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.17@tcp 192.168.203.17@tcp [ 1548.154564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.17@tcp1 192.168.203.17@tcp1 [ 1548.192292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.17@tcp -d 192.168.203.17@tcp -e network_timeout [ 1548.236518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -e network_timeout [ 1548.282308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -e network_timeout [ 1548.325721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -e network_timeout [ 1548.365698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 1548.397233] LNet: There was an unexpected network error while writing to 192.168.203.17: rc = -22 [ 1548.419484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1548.522484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1548.630114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1548.704700] LNet: 1 local NIs in recovery (showing 1): 192.168.203.17@tcp [ 1554.768257] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 10:11:21 (1787839881) [ 1554.976630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1555.004607] LNet: 77611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.011574] LNetError: 77611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.017466] LNet: Removed LNI 192.168.203.17@tcp [ 1556.082187] LNet: Removed LNI 192.168.203.17@tcp1 [ 1556.136089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1556.232072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1556.708151] LNet: 77653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.005690] Key type .llcrypt unregistered [ 1557.008741] Key type ._llcrypt unregistered [ 1569.822582] Key type ._llcrypt registered [ 1569.824664] Key type .llcrypt registered [ 1571.835906] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 10:11:36 (1787839896) [ 1572.460176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1572.485418] alg: No test for adler32 (adler32-zlib) [ 1573.375451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.006739] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1577.577362] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 10:11:42 (1787839902) [ 1577.636213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.668903] LNet: 78745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.688993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.741758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1577.761220] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1579.025535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1579.058704] LNet: Added LNI 192.168.203.17@tcp1 [8/256/0/180] [ 1580.475933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 1580.554475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp1 [ 1580.615143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp1 [ 1587.162236] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 10:11:53 (1787839913) [ 1588.929611] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg317-server [ 1590.314457] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 10:11:56 (1787839916) [ 1592.451642] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg317-server [ 1594.231498] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 10:12:00 (1787839920) [ 1596.215182] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg317-server [ 1597.803598] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 10:12:03 (1787839923) [ 1599.438706] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg317-server [ 1600.874287] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 10:12:07 (1787839927) [ 1602.264181] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg317-server [ 1603.559281] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 10:12:09 (1787839929) [ 1605.137870] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg317-server [ 1606.504953] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 10:12:12 (1787839932) [ 1608.157569] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg317-server [ 1609.560573] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 10:12:15 (1787839935) [ 1611.240858] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg317-server [ 1612.519632] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 10:12:18 (1787839938) [ 1613.921422] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg317-server [ 1615.498541] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 10:12:21 (1787839941) [ 1615.757323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.781890] LNet: 82628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1615.790413] LNetError: 82628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1615.800705] LNet: Removed LNI 192.168.203.17@tcp [ 1616.877869] LNet: Removed LNI 192.168.203.17@tcp1 [ 1616.891991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.933825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1616.968741] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1617.547491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1617.626469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 1617.813733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.840598] LNet: 82775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.849318] LNetError: 82775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.857511] LNetError: 82775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1617.870948] LNet: Removed LNI 192.168.203.17@tcp [ 1617.895471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.950693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1617.971207] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1618.676745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 1618.820674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1618.850472] LNetError: 82921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1618.867262] LNet: Removed LNI 192.168.203.17@tcp [ 1618.886940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.945858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1618.983902] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1619.749489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1619.818767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 1620.003496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.045862] LNet: 83068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1620.050795] LNet: 83068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1621.099396] LNet: Removed LNI 192.168.203.17@tcp [ 1621.129988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.179716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1621.205927] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1621.898647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 1623.164221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.187576] LNetError: 83222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.194425] LNetError: 83222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1623.210653] LNet: Removed LNI 192.168.203.17@tcp [ 1623.237141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.294914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1623.323871] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1623.949428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1624.057387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 1625.324696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.343339] LNet: 83378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.351179] LNet: 83378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1626.422270] LNet: Removed LNI 192.168.203.17@tcp [ 1626.448121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.512540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1626.562457] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1627.235776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 1627.471857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.510691] LNetError: 83524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1627.523976] LNetError: 83524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1627.565429] LNet: Removed LNI 192.168.203.17@tcp [ 1627.620814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.675674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1627.705251] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1628.362739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1628.457533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 1628.686426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1629.736258] LNet: Removed LNI 192.168.203.17@tcp [ 1629.779839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.847283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1629.876420] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1630.629619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 1630.989467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1632.037715] LNet: Removed LNI 192.168.203.17@tcp [ 1632.062603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.121635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1632.149345] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1632.940094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1632.997841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 1633.262932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.343559] LNet: Removed LNI 192.168.203.17@tcp [ 1634.367115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.438249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1634.468598] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1635.105770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 1635.330973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.356589] LNet: 84109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1635.363685] LNet: 84109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1636.389815] LNet: Removed LNI 192.168.203.17@tcp [ 1636.422812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.483459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1636.514464] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1637.343605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1637.437874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 1637.766132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.813469] LNetError: 84257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1637.823361] LNetError: 84257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1637.838745] LNet: Removed LNI 192.168.203.17@tcp [ 1637.872354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.932356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1637.975202] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1638.690625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 1638.948857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1640.052309] LNet: Removed LNI 192.168.203.17@tcp [ 1640.104201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.183640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1640.223513] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1641.003872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1641.060898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.106452] LNet: Removed LNI 192.168.203.17@tcp [ 1641.116871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.155594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1641.170858] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1641.745030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1641.843104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.884279] LNet: Removed LNI 192.168.203.17@tcp [ 1641.906848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.957314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1642.041103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.085841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.144528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1642.253265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.285791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.328730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1642.425940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.485374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.548722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1642.571537] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1643.305244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1643.344996] LNet: Removed LNI 192.168.203.17@tcp [ 1643.368299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1643.436697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1649.440643] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 10:12:55 (1787839975) [ 1649.630982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.669332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.716031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1649.736700] LNet: Added LNI 192.168.203.17@tcp231 [8/256/0/180] [ 1650.744945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1650.767721] LNet: Removed LNI 192.168.203.17@tcp231 [ 1650.786461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.890405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1650.917241] LNet: Added LNI 192.168.203.17@tcp231 [8/256/0/180] [ 1650.926054] LNet: 85569:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1651.437324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1651.475232] LNet: Removed LNI 192.168.203.17@tcp231 [ 1651.504783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1651.534251] LNet: Added LNI 192.168.203.17@tcp231 [8/256/0/180] [ 1652.664279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1652.683481] LNet: 85714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.689535] LNet: 85714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1652.704658] LNet: Removed LNI 192.168.203.17@tcp231 [ 1652.741648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.844761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1652.871289] LNet: Added LNI 192.168.203.17@tcp231 [8/256/0/0] [ 1652.884956] LNet: 85722:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1653.273966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1653.314441] LNet: Removed LNI 192.168.203.17@tcp231 [ 1653.346462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1653.369950] LNet: Added LNI 192.168.203.17@tcp231 [8/256/0/0] [ 1654.509768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.568749] LNetError: 85865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.576341] LNetError: 85865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1654.594844] LNet: Removed LNI 192.168.203.17@tcp231 [ 1654.625610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.712300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1654.747672] LNet: Added LNI 192.168.203.17@tcp231 [8/256/0/60] [ 1654.756250] LNet: 85873:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1655.140462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1655.190361] LNet: Removed LNI 192.168.203.17@tcp231 [ 1655.210783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1655.269785] LNet: Added LNI 192.168.203.17@tcp231 [8/256/0/60] [ 1656.598178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.652584] LNet: Removed LNI 192.168.203.17@tcp231 [ 1656.688749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.758932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1656.788282] LNet: Added LNI 192.168.203.17@tcp231 [8/256/0/180] [ 1656.792338] LNet: 86024:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1657.163183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1657.191564] LNet: Removed LNI 192.168.203.17@tcp231 [ 1657.219914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1657.255621] LNet: Added LNI 192.168.203.17@tcp231 [8/256/0/180] [ 1664.338987] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 10:13:10 (1787839990) [ 1664.641620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.668376] LNet: Removed LNI 192.168.203.17@tcp231 [ 1664.683222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.731286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1664.748413] LNet: Added LNI 192.168.203.17@tcp232 [8/256/0/180] [ 1665.483313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1665.529965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1665.575312] LNet: Removed LNI 192.168.203.17@tcp232 [ 1665.599398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1665.636777] LNet: Added LNI 192.168.203.17@tcp232 [8/256/0/180] [ 1666.300387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1666.329624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1666.375243] LNet: Removed LNI 192.168.203.17@tcp232 [ 1666.397657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1666.453288] LNet: Added LNI 192.168.203.17@tcp232 [8/256/0/180] [ 1667.097404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1667.146452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1667.183661] LNet: Removed LNI 192.168.203.17@tcp232 [ 1667.195187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1667.213328] LNet: Added LNI 192.168.203.17@tcp232 [8/256/0/180] [ 1667.770452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1667.807874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1667.836869] LNet: Removed LNI 192.168.203.17@tcp232 [ 1667.870282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1674.160328] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 10:13:20 (1787840000) [ 1675.336745] Key type .llcrypt unregistered [ 1675.339915] Key type ._llcrypt unregistered [ 1681.755366] Key type ._llcrypt registered [ 1681.760267] Key type .llcrypt registered [ 1682.030766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1682.070092] alg: No test for adler32 (adler32-zlib) [ 1683.025935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1683.064021] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1689.774465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.117@tcp [ 1698.666209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.712793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1698.729400] LNet: Added LNI 192.168.203.17@tcp1 [8/256/0/180] [ 1700.258266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_interrupt -r 2 [ 1700.290977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp1 -m GET -e local_interrupt -r 2 [ 1700.327812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_interrupt -r 2 [ 1700.381748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -m GET -e local_interrupt -r 2 [ 1700.437587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp -m GET -e local_interrupt -r 2 [ 1700.482175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp1 -m GET -e local_interrupt -r 2 [ 1700.541575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -m GET -e local_interrupt -r 2 [ 1700.589923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -m GET -e local_interrupt -r 2 [ 1700.654733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.17@tcp 192.168.203.117@tcp [ 1700.685539] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1700.692556] LNet: 1 local NIs in recovery (showing 1): 192.168.203.17@tcp [ 1701.022648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1701.051654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1701.084728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1701.124493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1701.165738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1701.220062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1701.253344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1701.294347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1701.320395] LNet: 88530:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1701.356856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1701.394618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1701.468204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1701.526200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1701.575304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1701.642150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1702.162773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_dropped -r 2 [ 1702.202311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp1 -m GET -e local_dropped -r 2 [ 1702.249829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_dropped -r 2 [ 1702.307827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -m GET -e local_dropped -r 2 [ 1702.356481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp -m GET -e local_dropped -r 2 [ 1702.407847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp1 -m GET -e local_dropped -r 2 [ 1702.461857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -m GET -e local_dropped -r 2 [ 1702.514714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -m GET -e local_dropped -r 2 [ 1702.576101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.17@tcp 192.168.203.117@tcp [ 1702.596981] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1702.856737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1702.897272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1702.934618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1702.980926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1703.028029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1703.070550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1703.137682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1703.170199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1703.208160] LNet: 88651:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1703.252872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1703.285893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1703.329837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1703.394353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1703.438827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1703.484782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1704.035664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_aborted -r 2 [ 1704.118774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp1 -m GET -e local_aborted -r 2 [ 1704.184227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_aborted -r 2 [ 1704.228257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -m GET -e local_aborted -r 2 [ 1704.261375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp -m GET -e local_aborted -r 2 [ 1704.328359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp1 -m GET -e local_aborted -r 2 [ 1704.382804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -m GET -e local_aborted -r 2 [ 1704.445478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -m GET -e local_aborted -r 2 [ 1704.532191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.17@tcp 192.168.203.117@tcp [ 1704.575694] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1704.858335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1704.894334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1704.939818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1704.985541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1705.026127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1705.060788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1705.095183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1705.120450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1705.141718] LNet: 88771:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1705.165370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1705.216428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1705.244867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1705.274675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1705.319528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1705.382533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1705.940456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_no_route -r 2 [ 1705.981349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp1 -m GET -e local_no_route -r 2 [ 1706.027324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_no_route -r 2 [ 1706.068558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -m GET -e local_no_route -r 2 [ 1706.099378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp -m GET -e local_no_route -r 2 [ 1706.134485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp1 -m GET -e local_no_route -r 2 [ 1706.176687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -m GET -e local_no_route -r 2 [ 1706.249645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -m GET -e local_no_route -r 2 [ 1706.291555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.17@tcp 192.168.203.117@tcp [ 1706.557976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1706.587452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1706.626811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1706.673272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1706.719239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1706.751669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1706.809302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1706.850856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1706.899904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1706.929813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1706.988180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1707.026222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1707.058822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1707.094549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1707.641624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e local_timeout -r 2 [ 1707.689423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp1 -m GET -e local_timeout -r 2 [ 1707.741793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e local_timeout -r 2 [ 1707.791948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -m GET -e local_timeout -r 2 [ 1707.835690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp -m GET -e local_timeout -r 2 [ 1707.888503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp1 -m GET -e local_timeout -r 2 [ 1707.936636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -m GET -e local_timeout -r 2 [ 1707.996601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -m GET -e local_timeout -r 2 [ 1708.058871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.17@tcp 192.168.203.117@tcp [ 1708.085243] LNet: There was an unexpected network error while writing to 192.168.203.117: rc = -22 [ 1708.099729] LNet: Skipped 1 previous similar message [ 1708.419440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1708.456291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1708.496984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1708.540692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1708.572529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1708.616181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1708.652493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1708.688564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1708.705512] LNet: 89012:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1708.708652] LNet: 89012:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1708.727346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1708.764126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1708.805373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1708.853652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1708.924174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1708.978076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1709.646826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp -m GET -e remote_dropped -r 2 [ 1709.698840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.117@tcp1 -m GET -e remote_dropped -r 2 [ 1709.756380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp -m GET -e remote_dropped -r 2 [ 1709.798699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp -d 192.168.203.17@tcp1 -m GET -e remote_dropped -r 2 [ 1709.856611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp -m GET -e remote_dropped -r 2 [ 1709.928551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.117@tcp1 -m GET -e remote_dropped -r 2 [ 1709.976412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp -m GET -e remote_dropped -r 2 [ 1710.038361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.17@tcp1 -d 192.168.203.17@tcp1 -m GET -e remote_dropped -r 2 [ 1710.070525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.17@tcp 192.168.203.117@tcp [ 1710.091380] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.117@tcp [ 1710.261955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1710.290354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1710.322635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1710.362460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1710.394735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1710.439903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1710.483137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1710.513586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1710.540263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1710.571710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1710.613614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1710.652995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1710.717628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1710.786645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1716.091395] LNet: 89210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.095887] LNetError: 89210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1716.109860] LNet: Removed LNI 192.168.203.17@tcp [ 1716.129638] LNet: Removed LNI 192.168.203.17@tcp1 [ 1716.524856] Key type .llcrypt unregistered [ 1716.533308] Key type ._llcrypt unregistered [ 1732.552305] Key type ._llcrypt registered [ 1732.554223] Key type .llcrypt registered [ 1733.722858] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 10:14:19 (1787840059) [ 1734.127604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1734.145615] alg: No test for adler32 (adler32-zlib) [ 1735.097270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.177702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1735.204676] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1736.126614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1736.157758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.17@tcp [ 1736.177805] LNetError: 90191:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.17@tcp to recovery [ 1736.271917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.17@tcp [ 1736.393474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1742.623689] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 10:14:28 (1787840068) [ 1742.739720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.765746] LNet: 90775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.776238] LNetError: 90775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1742.802194] LNet: Removed LNI 192.168.203.17@tcp [ 1742.831906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.958640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1742.993372] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1743.833393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1743.848187] LNet: Added LNI 192.168.203.17@tcp1 [8/256/0/180] [ 1744.970593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.17@tcp [ 1745.005452] LNetError: 91048:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.17@tcp to recovery [ 1745.951584] LNet: 1 local NIs in recovery (showing 1): 192.168.203.17@tcp [ 1745.953875] LNet: There was an unexpected network error while writing to 192.168.203.17: rc = -22 [ 1746.283937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.17@tcp [ 1754.181844] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 10:14:40 (1787840080) [ 1756.069817] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg317-server [ 1758.101873] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 10:14:43 (1787840083) [ 1759.726965] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg317-server [ 1761.477498] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 10:14:47 (1787840087) [ 1763.489327] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg317-server [ 1764.960513] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:14:51 (1787840091) [ 1765.123739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.156716] LNet: 92677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.168250] LNetError: 92677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1765.186790] LNet: Removed LNI 192.168.203.17@tcp [ 1765.199948] LNet: Removed LNI 192.168.203.17@tcp1 [ 1765.216618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.274938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1765.302514] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1766.133743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1766.210696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1766.257792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1766.328787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1766.359678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1766.395158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1772.148721] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 10:14:58 (1787840098) [ 1772.325415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1772.345435] LNet: 93402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1772.363388] LNetError: 93402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1772.379977] LNetError: 93402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1772.397871] LNet: Removed LNI 192.168.203.17@tcp [ 1772.424076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.475655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1772.495447] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1779.219696] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:15:05 (1787840105) [ 1779.430759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.446396] LNet: 94106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.450822] LNetError: 94106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.466526] LNet: Removed LNI 192.168.203.17@tcp [ 1779.492807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.548956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1779.605696] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1786.926312] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:15:12 (1787840112) [ 1788.391575] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1790.928685] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:15:16 (1787840116) [ 1791.414987] LNet: 95141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1791.435317] LNetError: 95141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1791.454171] LNet: Removed LNI 192.168.203.17@tcp [ 1792.127131] Key type .llcrypt unregistered [ 1792.132553] Key type ._llcrypt unregistered [ 1798.757981] Key type ._llcrypt registered [ 1798.761751] Key type .llcrypt registered [ 1798.997590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1799.022318] alg: No test for adler32 (adler32-zlib) [ 1800.072628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1800.128621] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1811.304418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.117@tcp [ 1817.466868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.117@tcp [ 1817.994354] LNet: 95568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.021019] LNetError: 95568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.054021] LNet: Removed LNI 192.168.203.17@tcp [ 1818.493418] Key type .llcrypt unregistered [ 1818.501034] Key type ._llcrypt unregistered [ 1829.434894] Key type ._llcrypt registered [ 1829.438587] Key type .llcrypt registered [ 1830.486346] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1832.000726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1834.175023] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:15:59 (1787840159) [ 1835.726954] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1837.730348] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:16:03 (1787840163) [ 1839.214517] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg317-server [ 1840.863739] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 10:16:07 (1787840167) [ 1842.800415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg317-server [ 1844.416460] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:16:10 (1787840170) [ 1846.037459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1846.067313] alg: No test for adler32 (adler32-zlib) [ 1847.094933] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 1847.340639] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1848.999146] Key type lgssc registered [ 1850.195497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1868.893666] Key type lgssc unregistered [ 1869.151861] LNet: 98747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.160564] LNetError: 98747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.175205] LNet: Removed LNI 192.168.203.17@tcp [ 1869.990616] Key type .llcrypt unregistered [ 1870.000965] Key type ._llcrypt unregistered [ 1886.220350] Key type ._llcrypt registered [ 1886.224570] Key type .llcrypt registered [ 1887.427564] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 10:16:53 (1787840213) [ 1887.813874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1887.832078] alg: No test for adler32 (adler32-zlib) [ 1888.689247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1888.739635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1888.775396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1888.807565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1888.841750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1895.756920] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 10:17:01 (1787840221) [ 1895.888396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1895.944850] LNet: 100159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.993249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1896.075474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1896.095510] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1897.263721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1897.309938] LNet: 100309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1897.326517] LNetError: 100309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1897.343815] LNet: Removed LNI 192.168.203.17@tcp [ 1897.373499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1897.478731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1897.514747] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1898.237976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1898.263420] LNetError: 100337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1898.289445] LNet: Removed LNI 192.168.203.17@tcp [ 1898.305871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.358524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1898.381323] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1905.966413] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 10:17:11 (1787840231) [ 1906.106198] LNet: 100918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1906.113461] LNet: 100918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1906.123387] LNetError: 100918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1906.143230] LNet: Removed LNI 192.168.203.17@tcp [ 1907.176442] Key type .llcrypt unregistered [ 1907.181454] Key type ._llcrypt unregistered [ 1914.967686] Key type ._llcrypt registered [ 1914.970425] Key type .llcrypt registered [ 1915.201603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1915.216750] alg: No test for adler32 (adler32-zlib) [ 1916.140540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.201442] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1916.205264] LNet: *** cfs_fail_loc=e003, val=100*** [ 1916.991590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 1917.024857] LNet: *** cfs_fail_loc=e003, val=100*** [ 1917.027819] LNet: Skipped 2 previous similar messages [ 1917.041641] Ping reply received with mapping: [ 1917.043943] found metadata reply NI[192.168.203.17@tcp] [ 1917.046423] 11 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1917.049452] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1917.055300] 5a 5a 5a 5a 5a 5a 5a 5a [ 1917.215825] LNet: 101329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.221265] LNetError: 101329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1917.239048] LNet: Removed LNI 192.168.203.17@tcp [ 1917.923622] Key type .llcrypt unregistered [ 1917.930335] Key type ._llcrypt unregistered [ 1929.353246] Key type ._llcrypt registered [ 1929.356488] Key type .llcrypt registered [ 1930.464919] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:17:36 (1787840256) [ 1930.986424] Key type .llcrypt unregistered [ 1930.991055] Key type ._llcrypt unregistered [ 1937.963259] Key type ._llcrypt registered [ 1937.967566] Key type .llcrypt registered [ 1938.433187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1938.487364] alg: No test for adler32 (adler32-zlib) [ 1939.751781] LNetError: 102386:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1940.195839] Key type .llcrypt unregistered [ 1940.197978] Key type ._llcrypt unregistered [ 1945.989412] Key type ._llcrypt registered [ 1945.991688] Key type .llcrypt registered [ 1947.551806] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 10:17:53 (1787840273) [ 1948.229300] Key type .llcrypt unregistered [ 1948.235444] Key type ._llcrypt unregistered [ 1954.628565] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1954.637043] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1957.306840] Key type ._llcrypt registered [ 1957.311438] Key type .llcrypt registered [ 1957.617683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1957.644338] alg: No test for adler32 (adler32-zlib) [ 1958.596580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.692737] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1968.928570] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 10:18:14 (1787840294) [ 1969.258235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1969.285795] LNet: 104175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.293880] LNetError: 104175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1969.318803] LNet: Removed LNI 10.1.2.3@tcp [ 1969.360509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1970.186820] Key type .llcrypt unregistered [ 1970.195325] Key type ._llcrypt unregistered [ 1976.268742] Key type ._llcrypt registered [ 1976.272267] Key type .llcrypt registered [ 1976.475789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1976.503653] alg: No test for adler32 (adler32-zlib) [ 1977.486771] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 1977.567704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 1977.733852] LNet: 104467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1977.741132] LNetError: 104467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1977.775549] LNet: Removed LNI 192.168.203.17@tcp [ 1978.496288] Key type .llcrypt unregistered [ 1978.502222] Key type ._llcrypt unregistered [ 1990.071791] Key type ._llcrypt registered [ 1990.074318] Key type .llcrypt registered [ 1991.291161] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 10:18:37 (1787840317) [ 1992.210299] Key type .llcrypt unregistered [ 1992.211922] Key type ._llcrypt unregistered [ 1999.554275] Key type ._llcrypt registered [ 1999.556083] Key type .llcrypt registered [ 1999.773204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1999.794416] alg: No test for adler32 (adler32-zlib) [ 2007.265400] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 10:18:53 (1787840333) [ 2007.412588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2007.473516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2007.555381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2007.587091] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2008.450849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2008.486680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2008.521461] LNet: Removed LNI 192.168.203.17@tcp [ 2008.843250] LNet: 106221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.309734] Key type .llcrypt unregistered [ 2009.311750] Key type ._llcrypt unregistered [ 2021.807582] Key type ._llcrypt registered [ 2021.810216] Key type .llcrypt registered [ 2023.344620] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 10:19:08 (1787840348) [ 2023.836238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2023.883278] alg: No test for adler32 (adler32-zlib) [ 2024.858878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2024.942182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2024.977642] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2025.760518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 2025.822509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.17@tcp [ 2025.852613] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2025.860536] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2025.866770] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2025, 0, 0 [ 2025.876255] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2025.882347] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2025.893417] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2025.899529] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2025, 0, 0 [ 2025.909633] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2025.915603] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2025.920057] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2025.930695] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2025, 0, 2075 [ 2025.934711] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2025.937612] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2025.941253] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2025.944878] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2025, 0, 2075 [ 2025.957235] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2032.126609] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 10:19:17 (1787840357) [ 2032.587380] LNet: 107761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.596544] LNetError: 107761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2033.638339] LNet: Removed LNI 192.168.203.17@tcp [ 2034.478478] Key type .llcrypt unregistered [ 2034.481383] Key type ._llcrypt unregistered [ 2041.171432] Key type ._llcrypt registered [ 2041.175330] Key type .llcrypt registered [ 2041.549810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2041.573292] alg: No test for adler32 (adler32-zlib) [ 2042.663930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2042.772499] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2051.260932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.117@tcp [ 2061.128735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.164897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2061.189694] LNet: Added LNI 192.168.203.17@tcp1 [8/256/0/180] [ 2062.075323] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2062.279417] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2062.281745] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2062.284547] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2064.625874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2064.653795] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2073.941540] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 10:20:00 (1787840400) [ 2074.072302] LNet: 109268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2074.079431] LNetError: 109268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2074.086894] LNet: Removed LNI 192.168.203.17@tcp [ 2075.103177] LNet: Removed LNI 192.168.203.17@tcp1 [ 2075.108953] LNetError: 109268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2075.117702] LNetError: 109268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2075.149792] LNet: Removed LNI 10.1.2.3@tcp99 [ 2075.828628] Key type .llcrypt unregistered [ 2075.831964] Key type ._llcrypt unregistered [ 2081.111462] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2081.116932] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2083.618267] Key type ._llcrypt registered [ 2083.620860] Key type .llcrypt registered [ 2083.766947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2083.780178] alg: No test for adler32 (adler32-zlib) [ 2084.659679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.699471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2084.727344] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2085.560470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2085.587724] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2087.817146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2087.870922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2087.935599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2087.951510] LNetError: 110023:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2087.979224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2088.018289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2088.532132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2088.641260] LNet: 110043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.647830] LNetError: 110043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2088.652706] LNet: Removed LNI 10.1.2.3@tcp [ 2089.704690] LNet: Removed LNI 192.168.203.17@tcp [ 2090.404427] Key type .llcrypt unregistered [ 2090.409130] Key type ._llcrypt unregistered [ 2102.465801] Key type ._llcrypt registered [ 2102.474204] Key type .llcrypt registered [ 2103.951648] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 10:20:29 (1787840429) [ 2104.445914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2104.463505] alg: No test for adler32 (adler32-zlib) [ 2105.403365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.543698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2105.570133] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2106.716852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 2106.848568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg317-client@tcp [ 2113.625612] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 10:20:39 (1787840439) [ 2113.781717] LNet: 111580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.787114] LNetError: 111580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.804504] LNet: Removed LNI 192.168.203.17@tcp [ 2114.408680] Key type .llcrypt unregistered [ 2114.411090] Key type ._llcrypt unregistered [ 2121.470096] Key type ._llcrypt registered [ 2121.473430] Key type .llcrypt registered [ 2121.679490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2121.706567] alg: No test for adler32 (adler32-zlib) [ 2122.724233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2122.782253] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2122.820227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2130.118730] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 10:20:56 (1787840456) [ 2131.892467] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2134.200492] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 10:20:59 (1787840459) [ 2136.372959] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2139.230726] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 10:21:03 (1787840463) [ 2139.412420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.502910] LNet: 112997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2139.525769] LNetError: 112997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2139.576838] LNet: Removed LNI 192.168.203.17@tcp [ 2139.611247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.711301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2139.765787] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2141.091732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2141.211219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2150.070565] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 10:21:16 (1787840476) [ 2150.308756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2150.359123] LNet: 113710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2150.378215] LNetError: 113710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2150.437531] LNet: Removed LNI 192.168.203.17@tcp [ 2150.473766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.605093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2150.735176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2151.727233] Key type .llcrypt unregistered [ 2151.732284] Key type ._llcrypt unregistered [ 2163.666444] Key type ._llcrypt registered [ 2163.669190] Key type .llcrypt registered [ 2164.950505] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 10:21:30 (1787840490) [ 2165.292249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2165.305814] alg: No test for adler32 (adler32-zlib) [ 2166.267219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2166.349413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2166.388683] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2167.186337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2167.249726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2173.905594] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 10:21:39 (1787840499) [ 2174.027731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2174.077025] LNet: 115267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2174.090550] LNetError: 115267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2174.121908] LNet: Removed LNI 192.168.203.17@tcp [ 2174.154109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2174.240030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2174.293109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2180.178568] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 10:21:46 (1787840506) [ 2180.393869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2180.431195] LNet: 115853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.450409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.506323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2181.035749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2181.089610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2181.166476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2181.490189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2181.567190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2181.637327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2181.651866] LNet: 115886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.665283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.731583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2182.162759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2182.211669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2182.255611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2182.610767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2182.641367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2188.176212] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 10:21:54 (1787840514) [ 2188.411093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2188.428936] LNet: 116486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2188.454428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2188.533295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2188.574680] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2189.212106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2189.236739] LNet: Added LNI 192.168.203.17@tcp2 [8/256/0/180] [ 2190.018581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2190.069302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2190.228934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2190.249037] LNetError: 116765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2191.263150] LNet: Removed LNI 192.168.203.17@tcp [ 2191.271134] LNetError: 116765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2191.291517] LNet: Removed LNI 192.168.203.17@tcp2 [ 2191.316448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2191.377749] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2191.388787] LNet: Added LNI 192.168.203.17@tcp2 [8/256/0/180] [ 2191.397440] LNet: 116772:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2191.942768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2192.147595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2193.183122] LNet: Removed LNI 192.168.203.17@tcp [ 2193.195469] LNet: Removed LNI 192.168.203.17@tcp2 [ 2193.214819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.257740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2193.280071] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2193.792972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2193.810499] LNet: Added LNI 192.168.203.17@tcp2 [8/256/0/180] [ 2194.512854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2194.554848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2194.615427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2194.782055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2194.801949] LNet: 117093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2194.810607] LNet: 117093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2194.821205] LNetError: 117093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2194.827092] LNetError: 117093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2194.832935] LNet: Removed LNI 192.168.203.17@tcp [ 2195.877683] LNet: Removed LNI 192.168.203.17@tcp2 [ 2195.918267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2195.994365] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2196.000039] LNet: Added LNI 192.168.203.17@tcp2 [8/256/0/180] [ 2196.004522] LNet: 117099:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2196.425513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2196.594386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2196.614255] LNet: Removed LNI 192.168.203.17@tcp [ 2197.670545] LNet: Removed LNI 192.168.203.17@tcp2 [ 2197.728053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2197.812564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2197.842992] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2198.444762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2198.462667] LNet: Added LNI 192.168.203.17@tcp2 [8/256/0/180] [ 2199.094253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2199.158506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2199.326436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2199.341319] LNetError: 117416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2199.345615] LNetError: 117416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2199.352350] LNet: Removed LNI 192.168.203.17@tcp [ 2199.361219] LNet: Removed LNI 192.168.203.17@tcp2 [ 2199.375557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2199.436423] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2199.442064] LNet: Added LNI 192.168.203.17@tcp2 [8/256/0/180] [ 2199.446549] LNet: 117422:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2199.921802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2200.130488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2201.183452] LNet: Removed LNI 192.168.203.17@tcp [ 2201.209411] LNet: Removed LNI 192.168.203.17@tcp2 [ 2201.262987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2201.356080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2201.383065] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2202.036949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2202.059082] LNet: Added LNI 192.168.203.17@tcp2 [8/256/0/180] [ 2203.062595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2203.118717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2203.180207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2203.392675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2203.417350] LNet: 117743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.423215] LNet: 117743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2204.447201] LNet: Removed LNI 192.168.203.17@tcp [ 2205.479064] LNet: Removed LNI 192.168.203.17@tcp2 [ 2205.541850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.654161] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2205.669416] LNet: Added LNI 192.168.203.17@tcp2 [8/256/0/180] [ 2205.673695] LNet: 117750:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2206.227437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2206.370793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2206.386793] LNet: Removed LNI 192.168.203.17@tcp [ 2207.464155] LNet: Removed LNI 192.168.203.17@tcp2 [ 2207.525203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2207.613288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2207.640698] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2208.254624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2208.270878] LNet: Added LNI 192.168.203.17@tcp2 [8/256/0/180] [ 2209.012685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2209.045871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2209.104820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2209.498395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2209.517407] LNetError: 118083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2209.520739] LNetError: 118083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2210.591130] LNet: Removed LNI 192.168.203.17@tcp [ 2210.614358] LNet: Removed LNI 192.168.203.17@tcp2 [ 2210.664696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.741550] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2210.753697] LNet: Added LNI 192.168.203.17@tcp2 [8/256/0/180] [ 2210.760590] LNet: 118089:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2211.309871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2211.673986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2212.768460] LNet: Removed LNI 192.168.203.17@tcp [ 2212.796986] LNet: Removed LNI 192.168.203.17@tcp2 [ 2212.821324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.895288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2212.930631] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2213.742177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2213.760653] LNet: Added LNI 192.168.203.17@tcp2 [8/256/0/180] [ 2214.628781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2214.683594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2214.719671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2214.819297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2215.157072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2216.223273] LNet: Removed LNI 192.168.203.17@tcp [ 2217.257347] LNet: Removed LNI 192.168.203.17@tcp2 [ 2217.296911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2217.397862] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2217.407705] LNet: Added LNI 192.168.203.17@tcp2 [8/256/0/180] [ 2217.960647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2224.720706] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2226.392716] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 10:22:32 (1787840552) [ 2226.501145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.538493] LNet: 119205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.546680] LNet: 119205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2226.562725] LNetError: 119205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2226.571663] LNetError: 119205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2226.586093] LNet: Removed LNI 192.168.203.17@tcp [ 2227.623165] LNet: Removed LNI 192.168.203.17@tcp2 [ 2227.647671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.909333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2227.927098] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2228.510397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2228.534274] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2229.358356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2229.385243] LNet: Removed LNI 10.1.2.3@tcp [ 2229.405082] LNet: Removed LNI 192.168.203.17@tcp [ 2229.426329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2229.497678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2229.526581] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2229.530738] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2229.893037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2229.912334] LNet: Removed LNI 192.168.203.17@tcp [ 2229.924305] LNet: Removed LNI 10.1.2.3@tcp [ 2235.216992] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 10:22:41 (1787840561) [ 2235.455382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2235.496494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.550942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2235.580086] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2236.194925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2236.226265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 2236.253350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 2236.278026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 2236.304649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 2236.360877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 2236.411568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 2236.452829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 2236.508262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 2236.577703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 2236.618422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.17@tcp [ 2236.810703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2242.262751] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 10:22:48 (1787840568) [ 2242.453212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2243.496667] LNet: Removed LNI 192.168.203.17@tcp [ 2243.519861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.606223] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2243.692262] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2243.695872] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2243.700719] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2244.913101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2244.937663] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2245.613639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2245.644070] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2247.577742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2247.615157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2247.680313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.17@tcp -m PUT -e local_timeout -r 1 [ 2247.735415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2247.798273] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2247.804845] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2248.029300] LNetError: 120879:0:(lib-move.c:4801:lnet_parse()) 192.168.203.17@tcp, src 192.168.203.17@tcp: Dropping PUT (error -108 looking up sender) [ 2249.064418] LNet: Removed LNI 10.1.2.3@tcp [ 2250.086084] LNet: Removed LNI 192.168.203.17@tcp [ 2250.418137] Key type .llcrypt unregistered [ 2250.420871] Key type ._llcrypt unregistered [ 2262.961708] Key type ._llcrypt registered [ 2262.965488] Key type .llcrypt registered [ 2264.157334] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 10:23:10 (1787840590) [ 2264.683450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2264.697841] alg: No test for adler32 (adler32-zlib) [ 2265.641663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2266.207776] LNet: 122198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2266.705326] Key type .llcrypt unregistered [ 2266.707032] Key type ._llcrypt unregistered [ 2274.266811] Key type ._llcrypt registered [ 2274.269578] Key type .llcrypt registered [ 2274.521772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2274.549484] alg: No test for adler32 (adler32-zlib) [ 2275.437629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2275.473838] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2283.835446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.117@tcp [ 2284.110594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2284.727076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.117@tcp [ 2284.923968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2284.951638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2284.980911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2285.002292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2285.027402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2285.053527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2285.071380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2285.108113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2285.152380] LNet: 122652:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2285.170479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2285.207872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2285.241995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2285.281426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2285.323917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2285.379113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2286.041778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2286.320314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2286.360577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2286.402606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2286.448415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2286.506782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2286.554233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2286.607584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2286.663756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2286.681820] LNet: 122770:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2286.719953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2286.751881] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.117@tcp [ 2286.777954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2286.813948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2286.848787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2286.896132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2286.930869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2287.474096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.117@tcp [ 2287.629887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2287.655068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2287.728825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2287.784784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2287.832108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2287.871601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2287.898521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2287.934655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2287.959767] LNet: 122887:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2287.979257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2288.038271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2288.095755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2288.152451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2288.226911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2288.285833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2288.980694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2289.191524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2289.209034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2289.238869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2289.275616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2289.307798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2289.336920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2289.379849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2289.408979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2289.442576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2289.475167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2289.515271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2289.568946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2289.624167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2289.655829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2290.005825] LNet: 123054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2290.014910] LNetError: 123054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2290.026926] LNet: Removed LNI 192.168.203.17@tcp [ 2290.412125] Key type .llcrypt unregistered [ 2290.420159] Key type ._llcrypt unregistered [ 2307.646750] Key type ._llcrypt registered [ 2307.653496] Key type .llcrypt registered [ 2309.214854] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:23:54 (1787840634) [ 2309.746048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2309.776805] alg: No test for adler32 (adler32-zlib) [ 2310.690085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.983939] LNet: 123942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.501574] Key type .llcrypt unregistered [ 2311.505948] Key type ._llcrypt unregistered [ 2317.632411] Key type ._llcrypt registered [ 2317.637899] Key type .llcrypt registered [ 2317.945560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2317.988952] alg: No test for adler32 (adler32-zlib) [ 2318.978404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2319.020291] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2326.481129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.117@tcp [ 2336.690557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2336.741900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2336.760461] LNet: Added LNI 192.168.203.17@tcp1 [8/256/0/180] [ 2337.982947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2338.853629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.117@tcp [ 2339.154742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2339.207577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2339.248341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2339.289459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2339.320846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2339.355233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2339.412174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2339.487166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2339.524635] LNet: 124600:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2339.551637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2339.552413] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.117@tcp [ 2339.600419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2339.668191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2339.739911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2339.801713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2339.843058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2340.772384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2341.025444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2341.058229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2341.115157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2341.146755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2341.185122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2341.216257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2341.257126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2341.296702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2341.323829] LNet: 124717:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2341.338317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2341.383705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2341.434109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2341.477842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2341.513310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2341.567509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2342.343108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.117@tcp [ 2342.485630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2342.512764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2342.531359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2342.558556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2342.590785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2342.624449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2342.678546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2342.718809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2342.734857] LNet: 124834:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2342.764655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2342.811070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2342.836719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2342.871647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2342.903848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2342.928415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2343.615212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2343.986488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2344.008128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2344.043284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2344.059922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2344.085318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2344.106682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2344.124784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2344.148399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2344.173888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2344.197853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2344.221695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2344.244157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2344.290862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2344.332652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2349.579519] LNet: 125037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2349.585252] LNetError: 125037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2350.623672] LNet: Removed LNI 192.168.203.17@tcp [ 2350.626146] LNetError: 125037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2350.666649] LNet: Removed LNI 192.168.203.17@tcp1 [ 2351.355765] Key type .llcrypt unregistered [ 2351.358245] Key type ._llcrypt unregistered [ 2372.747762] Key type ._llcrypt registered [ 2372.751315] Key type .llcrypt registered [ 2373.892145] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 10:24:59 (1787840699) [ 2374.215492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2374.229836] alg: No test for adler32 (adler32-zlib) [ 2375.142050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2375.208720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2375.234097] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2375.888056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2375.917560] LNet: Added LNI 192.168.203.17@tcp2 [8/256/0/180] [ 2376.694620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.17@tcp [ 2376.807336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2377.889719] LNet: Removed LNI 192.168.203.17@tcp [ 2383.016688] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 10:25:09 (1787840709) [ 2384.626300] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2386.046220] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 10:25:12 (1787840712) [ 2392.597566] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 10:25:18 (1787840718) [ 2399.164955] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 10:25:25 (1787840725) [ 2405.010927] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 10:25:31 (1787840731) [ 2410.506394] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 10:25:36 (1787840736) [ 2417.131960] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 10:25:43 (1787840743) [ 2424.339135] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 10:25:50 (1787840750) [ 2431.142814] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 10:25:57 (1787840757) [ 2437.318571] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 10:26:03 (1787840763) [ 2443.788143] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 10:26:09 (1787840769) [ 2449.729310] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 10:26:15 (1787840775) [ 2456.522983] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 10:26:22 (1787840782) [ 2463.990046] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 10:26:29 (1787840789) [ 2473.335149] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 10:26:38 (1787840798) [ 2483.440699] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 10:26:47 (1787840807) [ 2492.963841] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 10:26:59 (1787840819) [ 2499.930291] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 10:27:06 (1787840826) [ 2508.272559] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 10:27:13 (1787840833) [ 2508.640319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2508.675326] LNet: 136733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2508.697393] LNetError: 136733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2508.715704] LNet: Removed LNI 192.168.203.17@tcp2 [ 2508.745465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2508.894488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2508.912703] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2510.409605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2510.426468] LNet: 136875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2510.433890] LNetError: 136875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2510.456382] LNet: Removed LNI 192.168.203.17@tcp [ 2510.485444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2510.565456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2510.589101] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2518.785462] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 10:27:24 (1787840844) [ 2528.537548] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 10:27:34 (1787840854) [ 2539.150598] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 10:27:44 (1787840864) [ 2548.534508] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:27:54 (1787840874) [ 2555.381545] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:28:01 (1787840881) [ 2563.313589] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:28:09 (1787840889) [ 2572.759523] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:28:18 (1787840898) [ 2581.472187] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:28:27 (1787840907) [ 2590.380740] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:28:36 (1787840916) [ 2597.873105] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:28:43 (1787840923) [ 2605.724157] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:28:51 (1787840931) [ 2614.373686] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:28:59 (1787840939) [ 2622.186401] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2327 sec ========== 10:29:07 (1787840947) [ 2622.759859] LNet: 144344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2622.783202] LNetError: 144344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2622.800379] LNet: Removed LNI 192.168.203.17@tcp [ 2623.604661] Key type .llcrypt unregistered [ 2623.606376] Key type ._llcrypt unregistered [ 2638.143938] Key type ._llcrypt registered [ 2638.154645] Key type .llcrypt registered [ 2638.739811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2638.772907] alg: No test for adler32 (adler32-zlib) [ 2639.871352] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 2640.176785] LNet: Added LNI 192.168.203.17@tcp [8/256/0/180] [ 2641.847271] Key type lgssc registered [ 2642.906982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2706.215672] Lustre: Mounted lustre-client - version 2.17.57_95_g5b39979 [ 2711.295897] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2719.840673] Lustre: Unmounted lustre-client [ 2757.882698] Key type lgssc unregistered [ 2758.159476] LNet: 147306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2758.178064] LNetError: 147306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2758.199074] LNet: Removed LNI 192.168.203.17@tcp [ 2758.935154] Key type .llcrypt unregistered [ 2758.936984] Key type ._llcrypt unregistered