[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 456631402 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002364] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006858] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009008] pid_max: default: 32768 minimum: 301 [ 0.011062] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.014040] SELinux: Initializing. [ 0.015019] *** VALIDATE selinux *** [ 0.023116] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028772] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033117] *** VALIDATE tmpfs *** [ 0.035079] *** VALIDATE proc *** [ 0.036240] *** VALIDATE cgroup *** [ 0.037011] *** VALIDATE cgroup2 *** [ 0.038284] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039186] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042036] Spectre V2 : User space: Vulnerable [ 0.043014] Speculative Store Bypass: Vulnerable [ 0.046580] debug: unmapping init [mem 0xffffffffad859000-0xffffffffad860fff] [ 0.048177] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049526] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050021] ... version: 2 [ 0.051013] ... bit width: 48 [ 0.052014] ... generic registers: 4 [ 0.053013] ... value mask: 0000ffffffffffff [ 0.054028] ... max period: 00007fffffffffff [ 0.055013] ... fixed-purpose events: 3 [ 0.056019] ... event mask: 000000070000000f [ 0.057319] rcu: Hierarchical SRCU implementation. [ 0.059486] smp: Bringing up secondary CPUs ... [ 0.060583] x86: Booting SMP configuration: [ 0.061041] .... node #0, CPUs: #1 #2 #3 [ 0.064429] smp: Brought up 1 node, 4 CPUs [ 0.066011] smpboot: Max logical packages: 1 [ 0.067018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146156] node 0 deferred pages initialised in 77ms [ 0.149131] devtmpfs: initialized [ 0.150276] x86/mm: Memory block size: 128MB [ 0.152427] gcov: version magic: 0x41383552 [ 0.153603] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157065] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159277] pinctrl core: initialized pinctrl subsystem [ 0.161132] [ 0.161672] ************************************************************* [ 0.164013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167012] ** ** [ 0.170012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172012] ** ** [ 0.175011] ** This means that this kernel is built to expose internal ** [ 0.177014] ** IOMMU data structures, which may compromise security on ** [ 0.180013] ** your system. ** [ 0.183014] ** ** [ 0.185012] ** If you see this message and you are not debugging the ** [ 0.187010] ** kernel, report this immediately to your vendor! ** [ 0.190012] ** ** [ 0.192007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194014] ************************************************************* [ 0.195498] NET: Registered protocol family 16 [ 0.197414] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.200057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.206089] cpuidle: using governor menu [ 0.207665] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.212459] PCI: Using configuration type 1 for base access [ 0.214123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.224070] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.226038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.230150] cryptd: max_cpu_qlen set to 1000 [ 0.234275] ACPI: Added _OSI(Module Device) [ 0.235027] ACPI: Added _OSI(Processor Device) [ 0.237016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.238015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242977] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248361] ACPI: Interpreter enabled [ 0.250042] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251000] ACPI: Using IOAPIC for interrupt routing [ 0.251000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.256489] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.266797] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.281314] acpiphp: Slot [2] registered [ 0.283114] acpiphp: Slot [5] registered [ 0.284133] acpiphp: Slot [6] registered [ 0.285131] acpiphp: Slot [3] registered [ 0.287136] acpiphp: Slot [4] registered [ 0.290126] acpiphp: Slot [7] registered [ 0.291111] acpiphp: Slot [8] registered [ 0.293114] acpiphp: Slot [9] registered [ 0.294089] acpiphp: Slot [10] registered [ 0.296110] acpiphp: Slot [11] registered [ 0.297152] acpiphp: Slot [12] registered [ 0.299123] acpiphp: Slot [13] registered [ 0.300158] acpiphp: Slot [14] registered [ 0.302130] acpiphp: Slot [15] registered [ 0.304099] acpiphp: Slot [16] registered [ 0.305118] acpiphp: Slot [17] registered [ 0.307109] acpiphp: Slot [18] registered [ 0.308087] acpiphp: Slot [19] registered [ 0.310098] acpiphp: Slot [20] registered [ 0.311094] acpiphp: Slot [21] registered [ 0.314119] acpiphp: Slot [22] registered [ 0.315126] acpiphp: Slot [23] registered [ 0.317098] acpiphp: Slot [24] registered [ 0.319171] acpiphp: Slot [25] registered [ 0.320132] acpiphp: Slot [26] registered [ 0.322143] acpiphp: Slot [27] registered [ 0.323145] acpiphp: Slot [28] registered [ 0.325172] acpiphp: Slot [29] registered [ 0.327150] acpiphp: Slot [30] registered [ 0.329148] acpiphp: Slot [31] registered [ 0.330113] PCI host bridge to bus 0000:00 [ 0.332034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.335039] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.338035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.340035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.343041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.346040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.348254] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.351671] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.355365] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.363014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.368062] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.371020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.374020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.377018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.379571] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.382813] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.385045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.388650] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.392957] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.404023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.408024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.414626] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.419980] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.435020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.443391] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.451016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.462015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.481015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.495461] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.498421] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.502339] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.505392] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.508249] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.513131] iommu: Default domain type: Passthrough [ 0.515602] SCSI subsystem initialized [ 0.517457] ACPI: bus type USB registered [ 0.519393] usbcore: registered new interface driver usbfs [ 0.522156] usbcore: registered new interface driver hub [ 0.525138] usbcore: registered new device driver usb [ 0.527278] pps_core: LinuxPPS API ver. 1 registered [ 0.530021] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.533109] PTP clock support registered [ 0.536083] EDAC MC: Ver: 3.0.0 [ 0.538187] PCI: Using ACPI for IRQ routing [ 0.541027] NetLabel: Initializing [ 0.543014] NetLabel: domain hash size = 128 [ 0.544009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.546083] NetLabel: unlabeled traffic allowed by default [ 0.548186] vgaarb: loaded [ 0.550298] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.552026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.560310] clocksource: Switched to clocksource kvm-clock [ 0.661731] VFS: Disk quotas dquot_6.6.0 [ 0.663584] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.666299] *** VALIDATE ramfs *** [ 0.667614] *** VALIDATE hugetlbfs *** [ 0.669438] pnp: PnP ACPI init [ 0.672194] pnp: PnP ACPI: found 6 devices [ 0.691988] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.696159] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.698660] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.701104] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.703366] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.705473] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.708163] NET: Registered protocol family 2 [ 0.710413] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.714759] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.718702] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.724564] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.728195] TCP: Hash tables configured (established 65536 bind 65536) [ 0.731156] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.734015] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.736863] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.738673] NET: Registered protocol family 1 [ 0.740557] RPC: Registered named UNIX socket transport module. [ 0.743030] RPC: Registered udp transport module. [ 0.744935] RPC: Registered tcp transport module. [ 0.746649] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.748469] NET: Registered protocol family 44 [ 0.750279] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.751934] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.753228] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.755228] PCI: CLS 0 bytes, default 64 [ 0.756757] Unpacking initramfs... [ 2.131843] debug: unmapping init [mem 0xffff9d07bcc64000-0xffff9d07bffcffff] [ 2.136250] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.138141] software IO TLB: mapped [mem 0x00000000a3a00000-0x00000000a7a00000] (64MB) [ 2.141499] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.630132] Initialise system trusted keyrings [ 2.631493] Key type blacklist registered [ 2.633873] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.646924] zbud: loaded [ 2.650615] *** VALIDATE nfs *** [ 2.652278] *** VALIDATE nfs4 *** [ 2.654395] pstore: using deflate compression [ 2.658258] Platform Keyring initialized [ 2.786294] NET: Registered protocol family 38 [ 2.788418] Key type asymmetric registered [ 2.790100] Asymmetric key parser 'x509' registered [ 2.792129] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.795615] io scheduler mq-deadline registered [ 2.797634] io scheduler kyber registered [ 2.799486] io scheduler bfq registered [ 2.801631] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.804908] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.808033] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.811076] ACPI: Power Button [PWRF] [ 2.816288] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.822483] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.835525] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.863853] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.891701] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.895869] Non-volatile memory driver v1.3 [ 2.897483] Linux agpgart interface v0.103 [ 2.931031] virtio_blk virtio1: [vda] 146200 512-byte logical blocks (74.9 MB/71.4 MiB) [ 2.934305] vda: detected capacity change from 0 to 74854400 [ 2.948204] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.951246] vdb: detected capacity change from 0 to 1073741824 [ 2.957657] libphy: Fixed MDIO Bus: probed [ 2.962851] usbcore: registered new interface driver usbserial_generic [ 2.966712] usbserial: USB Serial support registered for generic [ 2.969418] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.974704] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.976863] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.979465] mousedev: PS/2 mouse device common for all mice [ 2.982781] rtc_cmos 00:05: RTC can wake from S4 [ 2.986178] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.990855] rtc_cmos 00:05: registered as rtc0 [ 2.993845] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.994963] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.001783] intel_pstate: CPU model not supported [ 3.003148] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.007148] hid: raw HID events driver (C) Jiri Kosina [ 3.008740] usbcore: registered new interface driver usbhid [ 3.010269] usbhid: USB HID core driver [ 3.012088] drop_monitor: Initializing network drop monitor service [ 3.014572] Initializing XFRM netlink socket [ 3.016593] NET: Registered protocol family 10 [ 3.019101] Segment Routing with IPv6 [ 3.020501] NET: Registered protocol family 17 [ 3.023345] mpls_gso: MPLS GSO support [ 3.029552] RAS: Correctable Errors collector initialized. [ 3.032023] AVX version of gcm_enc/dec engaged. [ 3.033732] AES CTR mode by8 optimization enabled [ 3.109675] sched_clock: Marking stable (3109656967, 0)->(3982577948, -872920981) [ 3.113141] registered taskstats version 1 [ 3.115141] Loading compiled-in X.509 certificates [ 3.117647] zswap: loaded using pool lzo/zbud [ 3.140697] Key type big_key registered [ 3.153426] Key type encrypted registered [ 3.155173] ima: No TPM chip found, activating TPM-bypass! [ 3.157657] ima: Allocated hash algorithm: sha1 [ 3.159567] ima: No architecture policies found [ 3.161092] evm: Initialising EVM extended attributes: [ 3.163310] evm: security.selinux [ 3.164782] evm: security.ima [ 3.166041] evm: security.capability [ 3.167570] evm: HMAC attrs: 0x1 [ 3.170043] rtc_cmos 00:05: setting system clock to 2026-08-26 08:40:37 UTC (1787733637) [ 3.176147] debug: unmapping init [mem 0xffffffffae803000-0xffffffffae9fffff] [ 3.178603] debug: unmapping init [mem 0xffffffffad582000-0xffffffffad858fff] [ 3.188151] Write protecting the kernel read-only data: 28672k [ 3.190959] debug: unmapping init [mem 0xffffffffabc03000-0xffffffffabdfffff] [ 3.193367] debug: unmapping init [mem 0xffffffffac514000-0xffffffffac5fffff] [ 3.223695] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.233824] systemd[1]: Detected virtualization kvm. [ 3.236120] systemd[1]: Detected architecture x86-64. [ 3.238318] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.266376] systemd[1]: No hostname configured. [ 3.267619] systemd[1]: Set hostname to . [ 3.269160] random: systemd: uninitialized urandom read (16 bytes read) [ 3.271309] systemd[1]: Initializing machine ID from random generator. [ 3.394175] random: systemd: uninitialized urandom read (16 bytes read) [ 3.396548] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.402260] random: systemd: uninitialized urandom read (16 bytes read) [ 3.404395] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.410459] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting Journal Service... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.028293] device-mapper: uevent: version 1.0.3 [ 4.031270] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.807243] virtio_net virtio0 ens2: renamed from eth0 [ 4.813678] scsi host0: ata_piix [ 4.844395] scsi host1: ata_piix [ 4.846466] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.851084] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.152537] dracut-initqueue[600]: [ 9.154579] random: fast init done RTNETLINK answers: File exists [ 9.672576] random: crng init done [ 9.673858] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.004101] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.009209] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.241994] SELinux: Disabled at runtime. [ 11.299374] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.308750] systemd[1]: Detected virtualization kvm. [ 11.310767] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.743706] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.746289] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.749919] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.753801] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.757487] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.764971] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.769051] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ 11.847238] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.182820] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.452521] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.511639] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.543906] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.559249] EDAC sbridge: Ver: 1.1.2 [ 13.538218] Key type dns_resolver registered [ 13.837772] NFS: Registering the id_resolver key type [ 13.840119] Key type id_resolver registered [ 13.842023] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg345-client login: [ 42.460852] libcfs: loading out-of-tree module taints kernel. [ 42.495176] Key type ._llcrypt registered [ 42.496994] Key type .llcrypt registered [ 42.827673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 42.836477] alg: No test for adler32 (adler32-zlib) [ 43.860878] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 44.357912] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 46.208242] Key type lgssc registered [ 48.053887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 156.491122] Lustre: Mounted lustre-client - version 2.17.57_80_g6de652b [ 161.299275] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 178.025283] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing check_logdir /tmp/testlogs/ [ 182.240268] Lustre: lustre-OST0000-osc-ffff9d081874c000: disconnect after 23s idle [ 182.350901] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing yml_node [ 186.310224] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 188.384635] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 190.612276] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 192.126362] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Aug 26 04:43:45 EDT 2026 [ 206.623670] Lustre: Unmounted lustre-client [ 231.549733] Key type lgssc unregistered [ 231.776794] LNet: 8456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 231.784679] LNetError: 8456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 231.802423] LNet: Removed LNI 192.168.203.45@tcp [ 232.554878] Key type .llcrypt unregistered [ 232.563856] Key type ._llcrypt unregistered [ 256.863415] Key type ._llcrypt registered [ 256.865882] Key type .llcrypt registered [ 257.137530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 257.161542] alg: No test for adler32 (adler32-zlib) [ 258.099363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 258.178593] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 258.221580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 262.299081] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 262.473617] LNet: 9341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 262.485242] LNetError: 9341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 262.538204] LNet: Removed LNI 192.168.203.45@tcp [ 263.290094] Key type .llcrypt unregistered [ 263.293135] Key type ._llcrypt unregistered [ 269.703576] Key type ._llcrypt registered [ 269.706196] Key type .llcrypt registered [ 271.038305] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 04:45:03 (1787733903) [ 271.365432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 271.377068] alg: No test for adler32 (adler32-zlib) [ 272.279929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.364169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 272.379928] LNet: 9760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 272.398325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.453987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 272.485218] LNet: 9767:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 278.893559] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 04:45:11 (1787733911) [ 278.988054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.009551] LNet: 10336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.036057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 279.179209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 285.656868] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 04:45:18 (1787733918) [ 285.739187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 285.784748] LNet: 10923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 285.810861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 285.899121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 292.499206] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 04:45:25 (1787733925) [ 292.631374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 292.652868] LNet: 11513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 292.671979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 292.747964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 298.301724] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 04:45:31 (1787733931) [ 298.552390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 298.573268] LNet: 12098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 298.592362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.643948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 299.086839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 304.936892] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 04:45:38 (1787733938) [ 305.146461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.196376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 305.281975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 305.573393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 305.933901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 311.468514] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 04:45:44 (1787733944) [ 311.591376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 311.632323] LNet: 13285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.640575] LNet: 13285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 311.660191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.734272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 315.485679] hrtimer: interrupt took 2767491 ns [ 318.075175] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 04:45:51 (1787733951) [ 318.209256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.284641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.408805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 318.446942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 318.751790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 318.799272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 319.189661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 319.232583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 319.631496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 319.691625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 319.990844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 320.016590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 320.377098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 320.428125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 320.902809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 320.931682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 321.603688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 321.675529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 322.171455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 322.233658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 322.569815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 322.603177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 322.973637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 322.996648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 323.371472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 323.447076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 323.800923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 323.841239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 329.791572] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 04:46:02 (1787733962) [ 329.994690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.009184] LNet: 14552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.012082] LNet: 14552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 330.025251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.061561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 330.110569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 336.636805] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 04:46:09 (1787733969) [ 336.707479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.751642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.851523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 336.916908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 342.622930] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 04:46:16 (1787733976) [ 342.686812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.723108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.815180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 342.843202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 348.047117] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 04:46:21 (1787733981) [ 348.153394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.214559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.300657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 348.323600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 354.321675] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 04:46:27 (1787733987) [ 354.403326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 354.464787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.524431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 354.567382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 359.717475] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 04:46:33 (1787733993) [ 359.865767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.898372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.943915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 359.965391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 364.747883] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 04:46:38 (1787733998) [ 364.898386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.911362] LNet: 18071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.916061] LNet: 18071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 364.930672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.029241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 365.389579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 371.382559] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 04:46:44 (1787734004) [ 371.503705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.540988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.731181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 371.931578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 380.076675] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 04:46:52 (1787734012) [ 380.332121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.388332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.533594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 380.800491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 389.264214] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 04:47:01 (1787734021) [ 389.352427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.430376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.147564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 390.339510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 396.670824] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 04:47:09 (1787734029) [ 396.988976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.053394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.145262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 397.201239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 403.786770] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 04:47:16 (1787734036) [ 404.018556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.049630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.140677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 404.233955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 410.186268] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 04:47:23 (1787734043) [ 410.491173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.525357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.646467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 416.416884] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 04:47:29 (1787734049) [ 416.517567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.574697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.677699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 422.515925] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 04:47:35 (1787734055) [ 422.600653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.670664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.777804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 423.226208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 429.223785] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 04:47:42 (1787734062) [ 429.327785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.349717] LNet: 23508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 429.361696] LNet: 23508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 429.390750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.489266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 429.539873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 435.381706] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 04:47:48 (1787734068) [ 435.492830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.549975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.625997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 435.655915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 441.170619] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 04:47:54 (1787734074) [ 441.306729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.381115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.474391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 441.535889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 447.511831] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 04:48:00 (1787734080) [ 447.607079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.639395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.736700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 447.782796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 453.369514] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 04:48:06 (1787734086) [ 453.542793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.613861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.697806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 453.746396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 453.763558] LNetError: 25873:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 453.835140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 453.868849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 459.317875] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 04:48:12 (1787734092) [ 459.590446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.643389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.742348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.tXcq0xezpA [ 459.802674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.tXcq0xezpA [ 459.869790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.tXcq0xezpA [ 465.852306] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 04:48:18 (1787734098) [ 465.959692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.984784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.043516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 466.102331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 466.162695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 472.054857] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 04:48:25 (1787734105) [ 472.285628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.343254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.411675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 472.444823] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 473.919824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 475.327363] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 477.216309] Key type lgssc registered [ 478.295380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 481.876095] LNetError: 27634:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.45@tcp99 has no available nets [ 484.348587] Key type lgssc unregistered [ 484.681977] LNetError: 28745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 484.698128] LNet: Removed LNI 192.168.203.45@tcp [ 485.368684] Key type .llcrypt unregistered [ 485.373795] Key type ._llcrypt unregistered [ 491.084773] Key type ._llcrypt registered [ 491.090665] Key type .llcrypt registered [ 492.248842] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 04:48:45 (1787734125) [ 492.684589] Key type .llcrypt unregistered [ 492.691955] Key type ._llcrypt unregistered [ 498.979747] Key type ._llcrypt registered [ 498.982904] Key type .llcrypt registered [ 499.182356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 499.206758] alg: No test for adler32 (adler32-zlib) [ 500.137180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.210770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 500.250541] LNet: 29666:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 500.258527] LNet: 29666:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 500.265518] LNet: 29666:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 500.277725] LNet: Message forwarding will be enabled in 2s [ 503.392255] LNet: Message forwarding enabled [ 503.695371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 503.721357] LNet: Message forwarding will stop in 2s to 6s [ 507.498595] LNet: Message forwarding disabled [ 508.136247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 508.151278] LNet: 29764:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 508.165349] LNet: 29764:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 508.170794] LNet: 29764:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 508.180979] LNet: Message forwarding will be enabled in 2s [ 508.393357] LNet: 29777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 509.130445] Key type .llcrypt unregistered [ 509.133721] Key type ._llcrypt unregistered [ 519.671163] Key type ._llcrypt registered [ 519.672893] Key type .llcrypt registered [ 520.687134] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 04:49:13 (1787734153) [ 520.978712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 520.995500] alg: No test for adler32 (adler32-zlib) [ 521.877889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.945505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 527.181887] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 04:49:20 (1787734160) [ 527.333610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.372588] LNet: 31184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.400378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.447805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 527.476950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 527.504676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 527.531656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 527.564759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 527.597251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 527.640652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 527.677282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 527.716596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 527.744431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 527.767386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 527.790886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 527.819705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 527.850515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 527.881025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 527.914452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 527.941491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 527.975911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 528.005379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 528.029508] LNetError: 31228:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 533.043766] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 04:49:26 (1787734166) [ 533.245297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.269938] LNet: 31799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 533.289074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.363944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 538.890228] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 04:49:32 (1787734172) [ 540.316586] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg345-server [ 542.233294] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 04:49:35 (1787734175) [ 542.390484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.419698] LNet: 32724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 542.469085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.535516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 542.565689] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 543.417636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[46-50]@tcp [ 543.464672] LNet: 32863:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 543.510817] LNetError: 32734:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.46@tcp is being used as a gateway but routing feature is not turned on [ 543.523331] LNetError: 32734:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.46@tcp has gone from up to down [ 550.145139] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 04:49:43 (1787734183) [ 551.765345] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg345-server [ 553.125456] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 04:49:46 (1787734186) [ 553.286143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.315293] LNet: 33799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 553.321664] LNetError: 33799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 554.344233] LNet: Removed LNI 192.168.203.45@tcp [ 554.394255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.458609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 554.495866] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 555.145625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[46-51/2]@tcp [ 555.170646] LNet: 33916:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 555.176609] LNet: 33916:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 555.196887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[46-51/2]@tcp [ 555.201582] LNetError: 33803:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.46@tcp is being used as a gateway but routing feature is not turned on [ 555.239134] LNetError: 33803:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 555.254542] LNetError: 33803:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.46@tcp has gone from up to down [ 555.268443] LNetError: 33803:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 560.982910] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 04:49:54 (1787734194) [ 561.099810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.129809] LNet: 34504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 561.149086] LNetError: 34504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 561.191449] LNet: Removed LNI 192.168.203.45@tcp [ 561.239366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.323648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 561.404532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.463898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.545639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 561.601686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.651879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.733768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 561.846742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.902868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.977743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 562.037960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.068732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.116627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 562.166774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.199954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.260732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 562.321400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.354975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.398203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 562.474919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.514574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.572509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 562.648774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.695367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.749396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 562.823647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.874632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.951650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 563.037989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.073850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.112665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 563.186156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.222753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.304821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 568.489708] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 04:50:01 (1787734201) [ 570.167708] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg345-server [ 571.489491] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 04:50:04 (1787734204) [ 572.937228] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg345-server [ 574.282360] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 04:50:07 (1787734207) [ 574.375392] LNet: 35903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.389812] LNet: 35903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 574.992534] Key type .llcrypt unregistered [ 574.995689] Key type ._llcrypt unregistered [ 580.380679] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 580.558460] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 580.566726] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 580.570845] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 583.071728] Key type ._llcrypt registered [ 583.078161] Key type .llcrypt registered [ 583.269578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 583.287306] alg: No test for adler32 (adler32-zlib) [ 584.215964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.286576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 584.309989] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 585.042436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 585.070424] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 586.975823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 586.992226] LNet: Removed LNI 10.1.2.3@tcp [ 587.114833] LNet: 36634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 587.119989] LNetError: 36634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 587.137197] LNet: Removed LNI 192.168.203.45@tcp [ 587.709478] Key type .llcrypt unregistered [ 587.712446] Key type ._llcrypt unregistered [ 597.782956] Key type ._llcrypt registered [ 597.785619] Key type .llcrypt registered [ 598.739404] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 04:50:31 (1787734231) [ 599.226456] Key type .llcrypt unregistered [ 599.232165] Key type ._llcrypt unregistered [ 603.806447] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 603.839979] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 603.844062] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 603.847833] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 606.336341] Key type ._llcrypt registered [ 606.338099] Key type .llcrypt registered [ 606.466300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 606.483389] alg: No test for adler32 (adler32-zlib) [ 607.394673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.457383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 607.484291] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 608.342750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 608.364826] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 610.760675] LNet: 38133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.768469] LNetError: 38133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 610.777486] LNet: Removed LNI 10.1.2.3@tcp [ 610.803949] LNet: Removed LNI 192.168.203.45@tcp [ 611.376490] Key type .llcrypt unregistered [ 611.384806] Key type ._llcrypt unregistered [ 624.112049] Key type ._llcrypt registered [ 624.113706] Key type .llcrypt registered [ 625.395732] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 04:50:58 (1787734258) [ 626.040342] Key type .llcrypt unregistered [ 626.044665] Key type ._llcrypt unregistered [ 632.758972] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 632.873552] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 632.880094] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 632.892757] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 635.362918] Key type ._llcrypt registered [ 635.365247] Key type .llcrypt registered [ 635.528026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 635.543667] alg: No test for adler32 (adler32-zlib) [ 636.383869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.416860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 636.427983] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 637.803736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 637.818649] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 639.175506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 639.203901] LNet: Removed LNI 10.1.2.3@tcp [ 639.222614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 639.253338] LNet: Removed LNI 10.1.2.31@tcp [ 639.273381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 639.293655] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 639.827487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 639.839569] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 640.599257] LNet: 40057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.605384] LNetError: 40057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 640.609916] LNet: Removed LNI 10.1.2.31@tcp [ 640.627613] LNet: Removed LNI 10.1.2.3@tcp [ 641.149966] Key type .llcrypt unregistered [ 641.151700] Key type ._llcrypt unregistered [ 649.093626] Key type ._llcrypt registered [ 649.095874] Key type .llcrypt registered [ 650.274968] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 04:51:23 (1787734283) [ 650.567836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 650.584566] alg: No test for adler32 (adler32-zlib) [ 651.479707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.541232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 651.562969] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 652.290655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.309525] LNet: 41013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.319534] LNetError: 41013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 652.340818] LNet: Removed LNI 192.168.203.45@tcp [ 652.376135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.411802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.45@tcp [ 652.427653] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 653.128995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.45@tcp [ 653.163733] LNet: Removed LNI 192.168.203.45@tcp [ 653.233737] LNet: 41148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.767710] Key type .llcrypt unregistered [ 653.769985] Key type ._llcrypt unregistered [ 663.320289] Key type ._llcrypt registered [ 663.322191] Key type .llcrypt registered [ 664.507266] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 04:51:37 (1787734297) [ 666.208505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg345-server [ 668.059719] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 04:51:40 (1787734300) [ 668.566309] Key type .llcrypt unregistered [ 668.570213] Key type ._llcrypt unregistered [ 674.503583] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 674.611338] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 674.616561] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 674.620341] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 677.149300] Key type ._llcrypt registered [ 677.154565] Key type .llcrypt registered [ 677.292646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 677.305920] alg: No test for adler32 (adler32-zlib) [ 678.220723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.274380] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 678.895202] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 680.617203] LNet: 42989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.622730] LNetError: 42989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 680.627746] LNet: Removed LNI 192.168.203.45@tcp1 [ 680.635328] LNet: Removed LNI 10.1.2.3@tcp2 [ 681.235394] Key type .llcrypt unregistered [ 681.238048] Key type ._llcrypt unregistered [ 690.507302] Key type ._llcrypt registered [ 690.511421] Key type .llcrypt registered [ 691.666458] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 04:52:04 (1787734324) [ 692.061491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 692.080822] alg: No test for adler32 (adler32-zlib) [ 693.042969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.104454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 693.143424] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 693.987455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.000944] LNet: 43963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.007167] LNetError: 43963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 694.028478] LNet: Removed LNI 192.168.203.45@tcp [ 694.065514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.186953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 694.224162] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 700.078768] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 04:52:13 (1787734333) [ 700.165944] LNet: 44545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.176455] LNetError: 44545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.195983] LNet: Removed LNI 192.168.203.45@tcp [ 700.751820] Key type .llcrypt unregistered [ 700.758429] Key type ._llcrypt unregistered [ 705.381196] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 705.550569] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 705.554466] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 705.563109] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 708.049250] Key type ._llcrypt registered [ 708.051376] Key type .llcrypt registered [ 708.174875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 708.185301] alg: No test for adler32 (adler32-zlib) [ 709.091857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.181581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 709.214767] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 709.888422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 709.916121] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 711.780155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.797741] LNet: 45300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.803596] LNetError: 45300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 711.808777] LNet: Removed LNI 10.1.2.3@tcp [ 711.820533] LNet: Removed LNI 192.168.203.45@tcp [ 711.839370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.879746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 711.899715] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 711.902878] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 712.231403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.254285] LNet: Removed LNI 10.1.2.3@tcp [ 712.273811] LNet: Removed LNI 192.168.203.45@tcp [ 712.299089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.361782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 712.398714] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 712.402174] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 712.818605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 712.853691] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 713.662979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.681171] LNet: 45490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.685410] LNet: 45490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 713.695156] LNetError: 45490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 713.701767] LNetError: 45490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 713.707776] LNet: Removed LNI 10.1.2.3@tcp [ 713.709729] LNet: Removed LNI 192.168.203.45@tcp [ 713.721632] LNet: Removed LNI 192.168.203.45@tcp2 [ 713.742181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.798154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 713.826858] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 713.831622] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 713.837730] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 718.630315] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 04:52:32 (1787734352) [ 718.793957] LNet: 46086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.800681] LNetError: 46086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 718.805066] LNetError: 46086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 718.810401] LNet: Removed LNI 10.1.2.3@tcp [ 718.812904] LNet: Removed LNI 192.168.203.45@tcp [ 718.820902] LNet: Removed LNI 192.168.203.45@tcp2 [ 719.237746] Key type .llcrypt unregistered [ 719.241985] Key type ._llcrypt unregistered [ 724.173751] Key type ._llcrypt registered [ 724.178069] Key type .llcrypt registered [ 724.320857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 724.340208] alg: No test for adler32 (adler32-zlib) [ 725.223655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.274409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 725.295923] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 725.904373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.918258] LNet: 46485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.923236] LNetError: 46485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.933765] LNet: Removed LNI 192.168.203.45@tcp [ 725.957181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.022620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 726.050425] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 726.427279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.441729] LNet: 46515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.446773] LNetError: 46515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 726.459867] LNet: Removed LNI 192.168.203.45@tcp [ 726.488752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.565331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 726.584456] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 726.915180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.948750] LNet: Removed LNI 192.168.203.45@tcp [ 726.965204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.014445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 727.036541] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 727.447023] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 727.529863] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 727.534988] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 727.539239] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 729.926681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.973816] LNet: 46625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.980898] LNet: 46625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 729.999411] LNetError: 46625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.003683] LNetError: 46625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 730.021825] LNet: Removed LNI 192.168.203.45@tcp [ 730.035513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.100741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 730.131161] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 734.890957] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 04:52:48 (1787734368) [ 734.944587] LNet: 47215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.950890] LNetError: 47215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.967485] LNet: Removed LNI 192.168.203.45@tcp [ 735.478408] Key type .llcrypt unregistered [ 735.480736] Key type ._llcrypt unregistered [ 739.751675] Key type ._llcrypt registered [ 739.756578] Key type .llcrypt registered [ 739.912594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 739.923206] alg: No test for adler32 (adler32-zlib) [ 740.818224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.872288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 740.889682] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 741.569428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.589982] LNet: 47623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.607599] LNetError: 47623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.632801] LNet: Removed LNI 192.168.203.45@tcp [ 741.648264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.720196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 741.742870] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 747.146193] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 04:53:00 (1787734380) [ 747.345279] LNet: 48214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 747.348184] LNetError: 48214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 747.369432] LNet: Removed LNI 192.168.203.45@tcp [ 748.019448] Key type .llcrypt unregistered [ 748.021879] Key type ._llcrypt unregistered [ 752.533027] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 752.733846] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 752.748143] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 752.765606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 755.412334] Key type ._llcrypt registered [ 755.419428] Key type .llcrypt registered [ 755.652701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 755.676171] alg: No test for adler32 (adler32-zlib) [ 756.784361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.902369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 756.934841] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 758.093853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.105982] LNet: 48668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.109896] LNetError: 48668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.128780] LNet: Removed LNI 192.168.203.45@tcp [ 758.143396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.197302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 758.228843] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 765.596394] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 04:53:18 (1787734398) [ 765.688774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.709494] LNet: 49261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.714840] LNetError: 49261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 765.723884] LNet: Removed LNI 192.168.203.45@tcp [ 765.735637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.852919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 771.523848] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 04:53:24 (1787734404) [ 771.683683] LNet: 49848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.181555] Key type .llcrypt unregistered [ 772.185732] Key type ._llcrypt unregistered [ 777.321932] Key type ._llcrypt registered [ 777.323578] Key type .llcrypt registered [ 777.479848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 777.502544] alg: No test for adler32 (adler32-zlib) [ 778.453339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.541225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 778.564119] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 779.438427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.455108] LNet: 50250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.461670] LNetError: 50250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.478362] LNet: Removed LNI 192.168.203.45@tcp [ 779.507941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.578549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 779.604684] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 785.111494] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 04:53:38 (1787734418) [ 785.195457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.217502] LNet: 50838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.224749] LNetError: 50838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.235096] LNet: Removed LNI 192.168.203.45@tcp [ 785.260310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.324480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 789.936766] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 04:53:43 (1787734423) [ 791.069082] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 792.245406] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 04:53:45 (1787734425) [ 793.503111] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 794.860266] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 04:53:48 (1787734428) [ 795.914160] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 797.162653] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 04:53:50 (1787734430) [ 798.210343] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 799.130337] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 04:53:52 (1787734432) [ 800.125614] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 801.097086] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 04:53:54 (1787734434) [ 801.215363] LNet: 52891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.593944] Key type .llcrypt unregistered [ 801.595528] Key type ._llcrypt unregistered [ 805.016037] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 805.134583] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 805.139488] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 805.142021] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 807.590482] Key type ._llcrypt registered [ 807.592424] Key type .llcrypt registered [ 807.752552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 807.764038] alg: No test for adler32 (adler32-zlib) [ 808.624407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.662750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 808.675900] LNet: Added LNI 192.168.203.45@tcp [8/512/0/180] [ 808.679704] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 808.684154] LNet: 53218:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 808.772705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.798566] LNet: 53237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.803723] LNetError: 53237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 808.808300] LNet: Removed LNI 10.1.2.3@tcp [ 808.810829] LNetError: 53237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.821263] LNet: Removed LNI 192.168.203.45@tcp [ 808.840526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.884324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 808.904329] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 808.939178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.953372] LNet: 53261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.958337] LNetError: 53261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.981040] LNet: Removed LNI 192.168.203.45@tcp [ 808.996777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.056090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 809.075635] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 809.079892] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 809.082730] LNet: 53269:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 809.169657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.182207] LNet: 53287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.185724] LNetError: 53287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 809.190968] LNet: Removed LNI 10.1.2.3@tcp [ 809.193187] LNetError: 53287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.205841] LNet: Removed LNI 192.168.203.45@tcp [ 809.237833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.306499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 809.342205] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 809.346590] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 809.349336] LNet: 53295:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 809.463659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.481682] LNet: 53321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.485578] LNetError: 53321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 809.493117] LNet: Removed LNI 10.1.2.3@tcp [ 809.497143] LNetError: 53321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.517527] LNet: Removed LNI 192.168.203.45@tcp [ 809.556595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.651935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 809.698696] LNet: Added LNI 192.168.203.45@tcp [8/256/0/182] [ 809.703317] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 809.844419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.862410] LNet: 53349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.868770] LNetError: 53349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 809.873973] LNet: Removed LNI 10.1.2.3@tcp [ 809.876826] LNetError: 53349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.892472] LNet: Removed LNI 192.168.203.45@tcp [ 809.922524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.995552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 810.015957] LNet: Added LNI 192.168.203.45@tcp [8/256/0/182] [ 810.026189] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 810.031341] LNet: 53357:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 810.100712] LNet: Removed LNI 10.1.2.3@tcp [ 810.110799] LNet: Removed LNI 192.168.203.45@tcp [ 810.625313] Key type .llcrypt unregistered [ 810.627810] Key type ._llcrypt unregistered [ 820.248828] Key type ._llcrypt registered [ 820.255390] Key type .llcrypt registered [ 821.585315] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 04:54:14 (1787734454) [ 823.410086] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 825.679065] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 04:54:18 (1787734458) [ 826.213290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 826.236601] alg: No test for adler32 (adler32-zlib) [ 827.183333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.272932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 827.294597] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 828.448101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.45 [ 828.482981] LNet: 54624:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 828.489950] LNet: Removed LNI 192.168.203.45@tcp [ 834.521256] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 04:54:27 (1787734467) [ 834.621855] LNet: 55187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.633333] LNetError: 55187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.652844] LNet: Removed LNI 192.168.203.45@tcp [ 835.370295] Key type .llcrypt unregistered [ 835.374437] Key type ._llcrypt unregistered [ 840.823788] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 841.033843] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 843.666261] Key type ._llcrypt registered [ 843.669581] Key type .llcrypt registered [ 843.823578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 843.847423] alg: No test for adler32 (adler32-zlib) [ 845.101914] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 849.788319] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 04:54:43 (1787734483) [ 849.963255] LNet: 56111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.969914] LNetError: 56111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 850.006254] LNet: Removed LNI 10.1.2.3@tcp [ 850.615967] Key type .llcrypt unregistered [ 850.621411] Key type ._llcrypt unregistered [ 855.839769] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 856.019439] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 856.025425] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 856.029354] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 858.632539] Key type ._llcrypt registered [ 858.635661] Key type .llcrypt registered [ 858.761701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 858.776399] alg: No test for adler32 (adler32-zlib) [ 859.697865] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 864.965822] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 04:54:58 (1787734498) [ 866.054139] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 867.588332] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 04:55:00 (1787734500) [ 867.741430] LNet: 57299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.748442] LNetError: 57299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 867.759338] LNet: Removed LNI 10.1.2.3@tcp [ 868.319567] Key type .llcrypt unregistered [ 868.325250] Key type ._llcrypt unregistered [ 873.567940] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 873.782887] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 873.787265] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 873.792210] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 876.409946] Key type ._llcrypt registered [ 876.411490] Key type .llcrypt registered [ 876.626935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 876.641271] alg: No test for adler32 (adler32-zlib) [ 877.572528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.655649] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 883.821792] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 04:55:17 (1787734517) [ 884.103745] LNet: 58549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.112996] LNetError: 58549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 884.127067] LNet: Removed LNI 10.1.2.3@tcp [ 884.572449] Key type .llcrypt unregistered [ 884.575113] Key type ._llcrypt unregistered [ 889.298971] Key type ._llcrypt registered [ 889.301723] Key type .llcrypt registered [ 889.490737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 889.510912] alg: No test for adler32 (adler32-zlib) [ 890.452760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 890.484019] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 896.707143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 897.233150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_interrupt -r 1 [ 897.259964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_interrupt -r 1 [ 897.289606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 897.303891] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 897.477544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 897.507653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 897.538121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 897.576213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 897.602324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 897.628493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 897.662133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 897.698364] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 897.712124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 897.743566] LNet: 58990:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 897.771911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 897.818985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 897.867704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 897.913042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 897.946450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 897.974486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 898.000078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 898.456473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_dropped -r 1 [ 898.496070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_dropped -r 1 [ 898.536689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 898.555597] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 898.757762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 898.784758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 898.815764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 898.845177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 898.888355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 898.927302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 898.969606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 899.005963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 899.032123] LNet: 59099:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 899.063809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 899.096756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 899.136760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 899.170686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 899.206599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 899.255098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 899.285978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 899.694670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_aborted -r 1 [ 899.736052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_aborted -r 1 [ 899.780473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 899.807862] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 899.813787] LNet: Skipped 1 previous similar message [ 899.990349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 900.032116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 900.063909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 900.094860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 900.129307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 900.169640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 900.202282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 900.234322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 900.255719] LNet: 59208:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 900.267895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 900.296063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 900.340324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 900.367772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 900.402852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 900.457726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 900.505990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 900.935641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_no_route -r 1 [ 900.965179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_no_route -r 1 [ 901.001631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 901.177363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 901.207068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 901.254849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 901.295936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 901.337344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 901.375642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 901.414970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 901.464866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 901.496717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 901.519698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 901.546966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 901.574629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 901.596632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 901.625154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 901.658306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 902.102335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_timeout -r 1 [ 902.147227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_timeout -r 1 [ 902.184788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 902.199487] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 902.204737] LNet: Skipped 1 previous similar message [ 902.371693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 902.394396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 902.415439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 902.439275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 902.468745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 902.492366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 902.520785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 902.550800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 902.565615] LNet: 59426:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 902.573623] LNet: 59426:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 902.591683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 902.615625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 902.650024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 902.678582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 902.706616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 902.733467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 902.762791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 903.103786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_error -r 1 [ 903.135732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_error -r 1 [ 903.159153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 903.333759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 903.357487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 903.382923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 903.414832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 903.443535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 903.464183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 903.493545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 903.520725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 903.546969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 903.574847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 903.617926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 903.656919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 903.690594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 903.725788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 903.757827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 904.001906] LNet: 59581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.011559] LNetError: 59581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.023854] LNet: Removed LNI 192.168.203.45@tcp [ 904.366873] Key type .llcrypt unregistered [ 904.370851] Key type ._llcrypt unregistered [ 917.554116] Key type ._llcrypt registered [ 917.559632] Key type .llcrypt registered [ 918.604780] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 04:55:51 (1787734551) [ 919.533268] Key type .llcrypt unregistered [ 919.535632] Key type ._llcrypt unregistered [ 925.033717] Key type ._llcrypt registered [ 925.036712] Key type .llcrypt registered [ 925.224040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 925.238732] alg: No test for adler32 (adler32-zlib) [ 926.117343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 926.150197] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 932.424495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 940.920247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.946318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 940.980671] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 942.182334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_interrupt -r 1 [ 942.208365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_interrupt -r 1 [ 942.233613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_interrupt -r 1 [ 942.260439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_interrupt -r 1 [ 942.290512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_interrupt -r 1 [ 942.319599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_interrupt -r 1 [ 942.349282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_interrupt -r 1 [ 942.375633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_interrupt -r 1 [ 942.408990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 942.426105] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 942.431422] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 942.602771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 942.633603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 942.661051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 942.686475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 942.722807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 942.758989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 942.797046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 942.833605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 942.849094] LNet: 61054:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 942.863286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 942.885495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 942.907421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 942.934095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 942.972178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 943.013903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 943.047757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 943.443639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_dropped -r 1 [ 943.465244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_dropped -r 1 [ 943.500390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_dropped -r 1 [ 943.541641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_dropped -r 1 [ 943.587364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_dropped -r 1 [ 943.628401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_dropped -r 1 [ 943.689687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_dropped -r 1 [ 943.724973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_dropped -r 1 [ 943.755967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 943.769511] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 943.775203] LNet: Skipped 2 previous similar messages [ 943.960570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 943.985356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 944.013135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 944.039499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 944.069832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 944.095203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 944.118448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 944.150604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 944.172509] LNet: 61175:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 944.192400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 944.225394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 944.247822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 944.271375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 944.303673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 944.338185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 944.377789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 944.877107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_aborted -r 1 [ 944.916616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_aborted -r 1 [ 944.944823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_aborted -r 1 [ 944.986401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_aborted -r 1 [ 945.017301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_aborted -r 1 [ 945.047633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_aborted -r 1 [ 945.076718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_aborted -r 1 [ 945.108591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_aborted -r 1 [ 945.134969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 945.152349] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 945.159308] LNet: Skipped 2 previous similar messages [ 945.336810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 945.369841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 945.399829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 945.417397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 945.441312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 945.467486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 945.490422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 945.517546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 945.534590] LNet: 61297:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 945.554571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 945.578424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 945.603724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 945.628637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 945.658859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 945.690236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 945.715820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 946.102079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_no_route -r 1 [ 946.133637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_no_route -r 1 [ 946.159684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_no_route -r 1 [ 946.182430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_no_route -r 1 [ 946.214419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_no_route -r 1 [ 946.239200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_no_route -r 1 [ 946.267479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_no_route -r 1 [ 946.294527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_no_route -r 1 [ 946.321811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 946.493946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 946.522322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 946.545730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 946.576365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 946.604794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 946.627983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 946.653850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 946.681935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 946.706437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 946.740240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 946.762317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 946.796145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 946.832911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 946.866724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 946.895321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 947.291867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_timeout -r 1 [ 947.314773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_timeout -r 1 [ 947.346745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_timeout -r 1 [ 947.384762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_timeout -r 1 [ 947.413882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_timeout -r 1 [ 947.450686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_timeout -r 1 [ 947.488850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_timeout -r 1 [ 947.520759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_timeout -r 1 [ 947.553983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 947.573466] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 947.577596] LNet: Skipped 5 previous similar messages [ 947.732956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 947.767322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 947.795175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 947.823771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 947.852981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 947.885553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 947.917658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 947.954071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 947.971563] LNet: 61539:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 947.979561] LNet: 61539:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 948.013818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 948.032984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 948.055001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 948.081252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 948.108882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 948.140347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 948.185318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 948.646849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_error -r 1 [ 948.675169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_error -r 1 [ 948.704373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_error -r 1 [ 948.747652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_error -r 1 [ 948.782190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_error -r 1 [ 948.818857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_error -r 1 [ 948.849993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_error -r 1 [ 948.895707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_error -r 1 [ 948.923931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 949.119192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 949.143418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 949.169272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 949.201649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 949.244788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 949.273740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 949.306109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 949.340370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 949.371161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 949.397849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 949.426205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 949.457841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 949.488450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 949.512517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 949.542688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 953.393729] LNet: 61740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.397657] LNetError: 61740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.401167] LNet: Removed LNI 192.168.203.45@tcp [ 953.408326] LNet: Removed LNI 192.168.203.45@tcp1 [ 953.654598] Key type .llcrypt unregistered [ 953.657730] Key type ._llcrypt unregistered [ 965.638050] Key type ._llcrypt registered [ 965.640060] Key type .llcrypt registered [ 966.455660] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 04:56:39 (1787734599) [ 966.971743] Key type .llcrypt unregistered [ 966.974575] Key type ._llcrypt unregistered [ 971.134632] Key type ._llcrypt registered [ 971.137291] Key type .llcrypt registered [ 971.269243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 971.281600] alg: No test for adler32 (adler32-zlib) [ 972.156548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 972.190986] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 977.753349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 978.282513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_dropped -r 1 [ 978.309982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_dropped -r 1 [ 978.348309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 978.372542] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 978.560582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 978.586628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 978.607254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 978.627865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 978.646262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 978.676216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 978.703225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 978.732132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 978.749946] LNet: 62992:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 978.764676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 978.789535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 978.810438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 978.833209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 978.858844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 978.885808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 978.910818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 979.324287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_error -r 1 [ 979.361453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_error -r 1 [ 979.384405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 979.410053] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 979.572640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 979.594663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 979.629932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 979.659925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 979.698931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 979.736883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 979.767012] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 979.791786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 979.806402] LNet: 63101:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 979.824359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 979.860286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 979.891510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 979.922500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 979.954867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 979.976850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 980.001977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 980.359861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_timeout -r 1 [ 980.400796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_timeout -r 1 [ 980.443944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 980.461891] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 980.684788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 980.724163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 980.755976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 980.779421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 980.803704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 980.826351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 980.853242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 980.883416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 980.908951] LNet: 63211:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 980.926038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 980.948811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 980.975944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 981.008381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 981.033446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 981.064696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 981.100883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 981.376428] LNet: 63256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.380641] LNetError: 63256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.396860] LNet: Removed LNI 192.168.203.45@tcp [ 981.694294] Key type .llcrypt unregistered [ 981.699149] Key type ._llcrypt unregistered [ 993.816866] Key type ._llcrypt registered [ 993.818724] Key type .llcrypt registered [ 994.915419] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 04:57:08 (1787734628) [ 995.522499] Key type .llcrypt unregistered [ 995.526936] Key type ._llcrypt unregistered [ 1000.595817] Key type ._llcrypt registered [ 1000.598685] Key type .llcrypt registered [ 1000.829783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1000.849069] alg: No test for adler32 (adler32-zlib) [ 1001.760312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1001.791940] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1008.450967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1016.251267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1016.306306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1016.325388] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1017.416397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_dropped -r 1 [ 1017.474747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e remote_dropped -r 1 [ 1017.511150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_dropped -r 1 [ 1017.541672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e remote_dropped -r 1 [ 1017.579936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e remote_dropped -r 1 [ 1017.615720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e remote_dropped -r 1 [ 1017.650837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e remote_dropped -r 1 [ 1017.673035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e remote_dropped -r 1 [ 1017.707983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1017.726476] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1017.732822] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.145@tcp [ 1017.894896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1017.922378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1017.953535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1017.989062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1018.043942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1018.074921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1018.104600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1018.133980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1018.178691] LNet: 64726:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1018.213404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1018.251678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1018.297077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1018.326212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1018.358289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1018.382489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1018.412536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1018.446470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1018.936563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_error -r 1 [ 1018.975740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e remote_error -r 1 [ 1019.027691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_error -r 1 [ 1019.084842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e remote_error -r 1 [ 1019.135809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e remote_error -r 1 [ 1019.180957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e remote_error -r 1 [ 1019.221278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e remote_error -r 1 [ 1019.275835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e remote_error -r 1 [ 1019.322852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1019.353754] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1019.358735] LNet: Skipped 2 previous similar messages [ 1019.510401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1019.534688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1019.559558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1019.583947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1019.610409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1019.630780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1019.649907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1019.673975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1019.688523] LNet: 64849:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1019.700255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1019.728658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1019.752377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1019.775840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1019.796863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1019.817459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1019.843147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1019.871337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1020.257938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_timeout -r 1 [ 1020.281822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e remote_timeout -r 1 [ 1020.302627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_timeout -r 1 [ 1020.344803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e remote_timeout -r 1 [ 1020.385053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e remote_timeout -r 1 [ 1020.421386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e remote_timeout -r 1 [ 1020.441827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e remote_timeout -r 1 [ 1020.469787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e remote_timeout -r 1 [ 1020.506650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1020.524647] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1020.670843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1020.695221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1020.717232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1020.740085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1020.763673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1020.780594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1020.800730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1020.824504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1020.844225] LNet: 64972:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1020.857603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1020.893939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1020.917775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1020.946919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1020.985330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1021.002754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1021.021675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1021.050085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1024.837477] LNet: 65054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.846202] LNetError: 65054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1025.888918] LNet: Removed LNI 192.168.203.45@tcp [ 1025.892659] LNetError: 65054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1025.927736] LNet: Removed LNI 192.168.203.45@tcp1 [ 1026.332673] Key type .llcrypt unregistered [ 1026.335342] Key type ._llcrypt unregistered [ 1042.598053] Key type ._llcrypt registered [ 1042.601326] Key type .llcrypt registered [ 1043.803319] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 04:57:56 (1787734676) [ 1045.501282] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1047.616043] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 04:58:00 (1787734680) [ 1048.718381] Key type .llcrypt unregistered [ 1048.720678] Key type ._llcrypt unregistered [ 1053.860325] Key type ._llcrypt registered [ 1053.861972] Key type .llcrypt registered [ 1054.029387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1054.043522] alg: No test for adler32 (adler32-zlib) [ 1054.911662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1054.926237] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1061.786367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1062.315248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e network_timeout -r 1 [ 1062.376867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e network_timeout -r 1 [ 1062.424403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1062.465433] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1062.586622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1062.606239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1062.640258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1062.664538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1062.688189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1062.706847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1062.743540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1062.766387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1062.789756] LNet: 66617:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1062.805531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1062.830046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1062.867822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1062.895217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1062.928943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1062.973846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1063.264253] LNet: 66661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.268828] LNetError: 66661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1063.283634] LNet: Removed LNI 192.168.203.45@tcp [ 1063.619824] Key type .llcrypt unregistered [ 1063.625209] Key type ._llcrypt unregistered [ 1077.061937] Key type ._llcrypt registered [ 1077.065335] Key type .llcrypt registered [ 1077.268844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1077.297408] alg: No test for adler32 (adler32-zlib) [ 1078.219847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1078.266127] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1084.821469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1092.277309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1092.350887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1092.388551] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1093.590751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e network_timeout -r 1 [ 1093.630609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e network_timeout -r 1 [ 1093.659848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e network_timeout -r 1 [ 1093.696561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e network_timeout -r 1 [ 1093.725536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e network_timeout -r 1 [ 1093.768870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e network_timeout -r 1 [ 1093.823387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e network_timeout -r 1 [ 1093.868780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e network_timeout -r 1 [ 1093.911973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1093.928573] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1094.094812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1094.113403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1094.137637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1094.160777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1094.197892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1094.220103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1094.248246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1094.278242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1094.300110] LNet: 67544:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1094.331234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1094.362972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1094.395400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1094.428739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1094.459224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1094.497162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1098.191221] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1098.535649] LNet: 67621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1098.554597] LNetError: 67621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1098.567530] LNet: Removed LNI 192.168.203.45@tcp [ 1098.585456] LNet: Removed LNI 192.168.203.45@tcp1 [ 1098.934475] Key type .llcrypt unregistered [ 1098.937071] Key type ._llcrypt unregistered [ 1113.409212] Key type ._llcrypt registered [ 1113.410776] Key type .llcrypt registered [ 1114.720492] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 04:59:07 (1787734747) [ 1115.089874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1115.109085] alg: No test for adler32 (adler32-zlib) [ 1116.010207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1116.060411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1116.086569] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1116.810795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1116.841431] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1117.736984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1117.828639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1117.864935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1117.887997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1117.910920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1117.941066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1117.962973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.45@tcp [ 1117.976503] LNetError: 68721:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.45@tcp to recovery [ 1118.112351] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1118.115810] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1120.161911] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1120.380455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1124.258565] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1125.394542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1125.482376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1125.573257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1125.609434] LNet: 68828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.615189] LNetError: 68828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.623278] LNet: Removed LNI 192.168.203.45@tcp [ 1125.646759] LNet: Removed LNI 192.168.203.45@tcp1 [ 1125.736610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1125.850623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1125.924085] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1127.006953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1127.039554] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1128.102772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1128.149905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1128.216752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1128.286161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1128.339419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1128.385152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1128.423067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1128.468769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.45@tcp [ 1128.492845] LNetError: 69101:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.45@tcp to recovery [ 1128.866784] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1132.390352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1135.012394] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1135.019289] LNet: Skipped 1 previous similar message [ 1139.568340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1139.612306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1139.686638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1147.065487] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 04:59:39 (1787734779) [ 1147.240774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1147.277075] LNet: 69808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1147.291216] LNetError: 69808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1147.297566] LNetError: 69808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1147.317465] LNet: Removed LNI 192.168.203.45@tcp [ 1147.328902] LNet: Removed LNI 192.168.203.45@tcp1 [ 1147.358653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1147.444368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1147.477157] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1148.240312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1148.260972] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1149.074837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1149.157025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1149.211292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1149.263716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1149.312821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1149.357513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1149.389704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.45@tcp --health 0 [ 1149.476285] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.45@tcp [ 1149.512504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1150.498834] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1150.505986] LNet: Skipped 1 previous similar message [ 1165.016394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1165.198574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1165.234787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.262770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.45@tcp --health 500 [ 1165.382799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1172.107114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1172.165754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1172.201405] LNet: 70300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.210480] LNetError: 70300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.224350] LNetError: 70300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1172.233177] LNet: Removed LNI 192.168.203.45@tcp [ 1172.245922] LNet: Removed LNI 192.168.203.45@tcp1 [ 1172.272464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1172.348733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1172.396643] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1173.112799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1173.136074] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1173.885814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1173.920088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.979598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1174.004761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1174.053437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1174.104561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1174.153688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1174.206320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.45@tcp --health 0 [ 1175.411021] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1175.419092] LNet: Skipped 3 previous similar messages [ 1175.565061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1186.803326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1186.844636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1186.896944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1193.719346] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 05:00:26 (1787734826) [ 1194.151864] LNet: 71357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1194.157953] LNetError: 71357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1194.164547] LNetError: 71357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1195.232139] LNet: Removed LNI 192.168.203.45@tcp [ 1195.242703] LNet: Removed LNI 192.168.203.45@tcp1 [ 1195.902847] Key type .llcrypt unregistered [ 1195.909509] Key type ._llcrypt unregistered [ 1201.542321] Key type ._llcrypt registered [ 1201.545169] Key type .llcrypt registered [ 1201.753587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1201.768837] alg: No test for adler32 (adler32-zlib) [ 1202.705931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1202.732969] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1211.770202] Lustre: DEBUG MARKER: Initial discovery [ 1211.837408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.145@tcp [ 1214.079386] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1214.135735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1214.177710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.145@tcp [ 1214.203987] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1214.234768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1214.279150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.145@tcp -r 1 -m GET -l 3 [ 1214.321525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.145@tcp -r 1 -m GET -e local_error [ 1214.363615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.145@tcp -r 1 -m PUT -l 6 [ 1215.008340] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1215.703955] Lustre: DEBUG MARKER: Force oleg345-client.virtnet to discover 192.168.203.145@tcp (in background) [ 1215.815718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.145@tcp [ 1217.181110] Lustre: DEBUG MARKER: Force oleg345-server to discover 192.168.203.45@tcp [ 1218.912675] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1219.389323] Lustre: DEBUG MARKER: Wait for 72121 [ 1220.755481] Lustre: DEBUG MARKER: Finished wait on 72121 [ 1221.005584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1221.039735] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1221.056829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1221.360292] LNet: 72579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.366410] LNetError: 72579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.385689] LNet: Removed LNI 192.168.203.45@tcp [ 1221.773305] Key type .llcrypt unregistered [ 1221.775197] Key type ._llcrypt unregistered [ 1238.091679] Key type ._llcrypt registered [ 1238.094334] Key type .llcrypt registered [ 1239.266085] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 05:01:12 (1787734872) [ 1239.832299] Key type .llcrypt unregistered [ 1239.836490] Key type ._llcrypt unregistered [ 1245.526596] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1245.699719] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1245.706271] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1245.710109] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1248.219468] Key type ._llcrypt registered [ 1248.221500] Key type .llcrypt registered [ 1248.476979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1248.486819] alg: No test for adler32 (adler32-zlib) [ 1249.449543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1249.529355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1249.557027] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1250.119460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1250.140693] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1257.851959] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 05:01:30 (1787734890) [ 1257.934749] LNet: 74663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1257.945837] LNetError: 74663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1257.956046] LNet: Removed LNI 10.1.2.3@tcp [ 1257.973124] LNet: Removed LNI 192.168.203.45@tcp [ 1258.710430] Key type .llcrypt unregistered [ 1258.716570] Key type ._llcrypt unregistered [ 1265.192830] Key type ._llcrypt registered [ 1265.198321] Key type .llcrypt registered [ 1265.360387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1265.371471] alg: No test for adler32 (adler32-zlib) [ 1266.300225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1266.367227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1266.400506] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1267.030393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1267.056237] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1267.927049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.45@tcp [ 1269.025854] LNetError: Unexpected error -22 connecting to 192.168.203.45@tcp at host 192.168.203.45:988 [ 1273.219090] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 05:01:46 (1787734906) [ 1273.297541] LNet: 75798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.306466] LNetError: 75798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1273.313324] LNet: Removed LNI 10.1.2.3@tcp [ 1273.327632] LNet: Removed LNI 192.168.203.45@tcp [ 1273.915861] Key type .llcrypt unregistered [ 1273.921177] Key type ._llcrypt unregistered [ 1279.633623] Key type ._llcrypt registered [ 1279.639434] Key type .llcrypt registered [ 1279.879463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1279.908357] alg: No test for adler32 (adler32-zlib) [ 1280.819955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1280.872972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1280.914152] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1281.689912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1281.709907] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 1282.790342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.45@tcp1 --nid 192.168.203.45@tcp2 [ 1286.888934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp1 192.168.203.45@tcp2 [ 1286.906662] LNetError: 76646:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.45@tcp1 to 192.168.203.45@tcp2 - Route table may be misconfigured [ 1286.927621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp2 192.168.203.45@tcp1 [ 1292.729669] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 05:02:05 (1787734925) [ 1292.854958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1292.885751] LNet: 77214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1292.903070] LNetError: 77214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1292.912843] LNet: Removed LNI 192.168.203.45@tcp1 [ 1293.930554] LNet: Removed LNI 192.168.203.45@tcp2 [ 1293.963602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1294.017853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1294.060788] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1294.693643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1294.712893] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1295.477908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1295.546223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.45@tcp [ 1295.578895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp1 192.168.203.45@tcp1 [ 1295.625028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.45@tcp -d 192.168.203.45@tcp -e network_timeout [ 1295.679902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -e network_timeout [ 1295.736376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -e network_timeout [ 1295.801690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -e network_timeout [ 1295.853182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1295.878321] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1295.903733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1296.001302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1296.032610] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1296.117108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1301.582825] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 05:02:14 (1787734934) [ 1301.731377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1301.766690] LNet: 78078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.780907] LNetError: 78078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1301.792700] LNetError: 78078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1301.804664] LNet: Removed LNI 192.168.203.45@tcp [ 1302.832680] LNet: Removed LNI 192.168.203.45@tcp1 [ 1302.865677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1302.942580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1303.249484] LNet: 78121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1303.682714] Key type .llcrypt unregistered [ 1303.687505] Key type ._llcrypt unregistered [ 1314.301047] Key type ._llcrypt registered [ 1314.307199] Key type .llcrypt registered [ 1315.812204] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 05:02:28 (1787734948) [ 1316.104683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1316.114985] alg: No test for adler32 (adler32-zlib) [ 1317.055655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1318.470737] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1320.247280] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 05:02:33 (1787734953) [ 1320.429314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1320.445912] LNet: 79219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1320.460485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1320.499987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1320.518087] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1321.208871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1321.230599] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1322.159076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1322.214810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp1 [ 1322.288447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp1 [ 1327.775629] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 05:02:40 (1787734960) [ 1329.413523] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg345-server [ 1330.735249] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 05:02:43 (1787734963) [ 1332.322500] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg345-server [ 1333.680252] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 05:02:46 (1787734966) [ 1335.254855] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg345-server [ 1336.761277] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 05:02:49 (1787734969) [ 1338.324876] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg345-server [ 1339.845417] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 05:02:52 (1787734972) [ 1341.304980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg345-server [ 1342.899821] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 05:02:55 (1787734975) [ 1344.487535] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg345-server [ 1345.845443] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 05:02:59 (1787734979) [ 1347.715158] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg345-server [ 1349.352397] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 05:03:02 (1787734982) [ 1351.076912] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg345-server [ 1352.430930] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 05:03:05 (1787734985) [ 1354.072664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg345-server [ 1355.455408] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 05:03:08 (1787734988) [ 1355.533509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1355.553382] LNet: 83089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1355.564097] LNetError: 83089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1355.570257] LNet: Removed LNI 192.168.203.45@tcp [ 1356.582929] LNet: Removed LNI 192.168.203.45@tcp1 [ 1356.613946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1356.684919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1356.708558] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1357.305699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1357.392177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1357.526243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1357.548916] LNet: 83243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.558631] LNetError: 83243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.567182] LNetError: 83243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1357.585541] LNet: Removed LNI 192.168.203.45@tcp [ 1357.628437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.704218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1357.738670] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1358.383265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1358.557369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1358.582167] LNetError: 83388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1359.654925] LNet: Removed LNI 192.168.203.45@tcp [ 1359.682792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1359.730439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1359.766225] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1360.350759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1360.412993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1360.601557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1360.636393] LNet: 83535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1360.644164] LNet: 83535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1360.657352] LNetError: 83535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.672580] LNet: Removed LNI 192.168.203.45@tcp [ 1360.704203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.769142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1360.794610] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1361.363239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1361.533716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1362.618154] LNet: Removed LNI 192.168.203.45@tcp [ 1362.664472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1362.727560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1362.755687] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1363.319389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1363.391751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1363.561370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1364.653300] LNet: Removed LNI 192.168.203.45@tcp [ 1364.696716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1364.758787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1364.805389] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1365.411814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1365.662829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1365.683617] LNet: 83973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.693131] LNet: 83973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1365.701144] LNetError: 83973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1365.706235] LNetError: 83973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1365.721540] LNet: Removed LNI 192.168.203.45@tcp [ 1365.743527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1365.794305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1365.812146] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1366.451456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1366.533174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1366.763854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1367.861722] LNet: Removed LNI 192.168.203.45@tcp [ 1367.895087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1367.996622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1368.024441] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1368.811097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1369.053763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1370.090099] LNet: Removed LNI 192.168.203.45@tcp [ 1370.120320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1370.182020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1370.212400] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1370.838876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1370.960938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1371.258679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1372.326419] LNet: Removed LNI 192.168.203.45@tcp [ 1372.367867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1372.432632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1372.460741] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1373.276995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1373.617713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1374.699984] LNet: Removed LNI 192.168.203.45@tcp [ 1374.744150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.834747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1374.868385] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1375.493381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1375.557471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1377.060771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1377.094888] LNet: 84715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.099511] LNet: 84715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1377.110168] LNetError: 84715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.115408] LNetError: 84715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1378.150877] LNet: Removed LNI 192.168.203.45@tcp [ 1378.186849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1378.287735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1378.325261] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1378.985964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1379.235491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1380.333893] LNet: Removed LNI 192.168.203.45@tcp [ 1380.359585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1380.422681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1380.444678] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1381.096489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1381.162139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1381.195982] LNet: Removed LNI 192.168.203.45@tcp [ 1381.219611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.267780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1381.290511] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1381.928680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1382.008610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1382.034132] LNet: Removed LNI 192.168.203.45@tcp [ 1382.055684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.111926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1382.168967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1382.192518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.233196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1382.295981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1382.321167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.386593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1382.483615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1382.543160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.610102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1382.641475] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1383.390653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1383.418896] LNet: Removed LNI 192.168.203.45@tcp [ 1383.438833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1383.502184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1388.793884] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 05:03:41 (1787735021) [ 1388.879228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1388.910385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1388.951296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1388.968410] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/180] [ 1389.994417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1390.027903] LNet: Removed LNI 192.168.203.45@tcp231 [ 1390.065622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1390.189982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1390.241915] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/180] [ 1390.246325] LNet: 86027:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1390.818186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1390.878545] LNet: Removed LNI 192.168.203.45@tcp231 [ 1390.898325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1390.931433] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/180] [ 1392.221157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1392.265657] LNet: Removed LNI 192.168.203.45@tcp231 [ 1392.313684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1392.375714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1392.400236] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/0] [ 1392.408776] LNet: 86179:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1392.687864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1392.733682] LNet: Removed LNI 192.168.203.45@tcp231 [ 1392.749522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1392.775065] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/0] [ 1394.378189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1394.438315] LNet: 86322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1394.455357] LNet: 86322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1394.482048] LNetError: 86322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1394.496725] LNetError: 86322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1394.537939] LNet: Removed LNI 192.168.203.45@tcp231 [ 1394.562701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1394.629107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1394.651587] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/60] [ 1394.657164] LNet: 86330:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1394.907311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1394.949682] LNet: Removed LNI 192.168.203.45@tcp231 [ 1394.974432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1395.014681] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/60] [ 1396.159168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1396.182689] LNet: Removed LNI 192.168.203.45@tcp231 [ 1396.198131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.256910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1396.272822] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/180] [ 1396.458543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1396.486242] LNet: Removed LNI 192.168.203.45@tcp231 [ 1396.499848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1396.545395] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/180] [ 1406.034815] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 05:03:58 (1787735038) [ 1406.148797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1406.189804] LNet: Removed LNI 192.168.203.45@tcp231 [ 1406.258299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1406.455625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1406.485560] LNet: Added LNI 192.168.203.45@tcp232 [8/256/0/180] [ 1407.651885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1407.743910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1407.827068] LNet: Removed LNI 192.168.203.45@tcp232 [ 1407.859251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1407.966367] LNet: Added LNI 192.168.203.45@tcp232 [8/256/0/180] [ 1409.213769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1409.317748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1409.366518] LNet: Removed LNI 192.168.203.45@tcp232 [ 1409.398997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1409.457348] LNet: Added LNI 192.168.203.45@tcp232 [8/256/0/180] [ 1410.573476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1410.612426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1410.643906] LNet: Removed LNI 192.168.203.45@tcp232 [ 1410.666363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1410.692800] LNet: Added LNI 192.168.203.45@tcp232 [8/256/0/180] [ 1411.514296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1411.550877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1411.586557] LNet: Removed LNI 192.168.203.45@tcp232 [ 1411.597692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1418.247496] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 05:04:11 (1787735051) [ 1419.785870] Key type .llcrypt unregistered [ 1419.797597] Key type ._llcrypt unregistered [ 1426.093505] Key type ._llcrypt registered [ 1426.103152] Key type .llcrypt registered [ 1426.395646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1426.416644] alg: No test for adler32 (adler32-zlib) [ 1427.427569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1427.458792] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1434.842283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1443.857807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1443.941587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1443.985264] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1445.545499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_interrupt -r 2 [ 1445.578492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_interrupt -r 2 [ 1445.607969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_interrupt -r 2 [ 1445.657195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_interrupt -r 2 [ 1445.708307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_interrupt -r 2 [ 1445.785500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_interrupt -r 2 [ 1445.851919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_interrupt -r 2 [ 1445.900356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_interrupt -r 2 [ 1445.996686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1446.027740] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1446.039669] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1446.513656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1446.563590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1446.616408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1446.659798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1446.722120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1446.757741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1446.786489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1446.839656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1446.889987] LNet: 88987:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1446.911986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1446.983858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1447.014190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1447.046652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1447.081575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1447.127186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1448.130679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_dropped -r 2 [ 1448.178935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_dropped -r 2 [ 1448.235799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_dropped -r 2 [ 1448.293454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_dropped -r 2 [ 1448.358484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_dropped -r 2 [ 1448.422707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_dropped -r 2 [ 1448.505706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_dropped -r 2 [ 1448.594762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_dropped -r 2 [ 1448.644318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1448.698446] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1449.102771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1449.163628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1449.223981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1449.257631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1449.300653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1449.342216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1449.393903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1449.467733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1449.496787] LNet: 89107:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1449.533234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1449.578329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1449.634578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1449.724458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1449.816694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1449.894780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1450.786311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_aborted -r 2 [ 1450.845024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_aborted -r 2 [ 1450.936278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_aborted -r 2 [ 1450.993761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_aborted -r 2 [ 1451.071179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_aborted -r 2 [ 1451.141981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_aborted -r 2 [ 1451.225777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_aborted -r 2 [ 1451.334587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_aborted -r 2 [ 1451.400683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1451.433953] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1451.818603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1451.853671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1451.893247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1451.948298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1452.017613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1452.072830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1452.129399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1452.168667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1452.194345] LNet: 89228:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1452.238448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1452.292294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1452.336456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1452.392859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1452.442234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1452.481815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1452.890886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_no_route -r 2 [ 1452.929931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_no_route -r 2 [ 1452.968875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_no_route -r 2 [ 1453.002694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_no_route -r 2 [ 1453.049416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_no_route -r 2 [ 1453.090481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_no_route -r 2 [ 1453.136400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_no_route -r 2 [ 1453.196302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_no_route -r 2 [ 1453.243201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1453.456953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1453.488420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1453.519502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1453.546452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1453.575963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1453.617307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1453.639963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1453.658144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1453.692193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1453.730598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1453.782446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1453.821732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1453.873186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1453.908398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1454.479692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_timeout -r 2 [ 1454.514871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_timeout -r 2 [ 1454.545376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_timeout -r 2 [ 1454.587262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_timeout -r 2 [ 1454.633785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_timeout -r 2 [ 1454.664856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_timeout -r 2 [ 1454.698139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_timeout -r 2 [ 1454.721763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_timeout -r 2 [ 1454.742970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1454.763936] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1454.773874] LNet: Skipped 1 previous similar message [ 1455.032618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1455.054360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1455.097593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1455.144548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1455.176985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1455.207919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1455.274228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1455.312712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1455.344774] LNet: 89468:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1455.353324] LNet: 89468:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1455.381260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1455.433705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1455.492963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1455.545361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1455.609896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1455.704551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1456.287436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_dropped -r 2 [ 1456.319537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e remote_dropped -r 2 [ 1456.357231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_dropped -r 2 [ 1456.388622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e remote_dropped -r 2 [ 1456.428507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e remote_dropped -r 2 [ 1456.459945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e remote_dropped -r 2 [ 1456.501039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e remote_dropped -r 2 [ 1456.541326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e remote_dropped -r 2 [ 1456.594906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1456.619404] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.145@tcp [ 1456.856276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1456.894455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1456.938846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1456.978541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1457.024077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1457.064217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1457.116510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1457.156746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1457.194413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1457.215728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1457.253544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1457.276375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1457.308619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1457.341676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1463.252994] LNet: 89666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.258341] LNetError: 89666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1463.267779] LNet: Removed LNI 192.168.203.45@tcp [ 1463.276921] LNet: Removed LNI 192.168.203.45@tcp1 [ 1463.621678] Key type .llcrypt unregistered [ 1463.625711] Key type ._llcrypt unregistered [ 1475.086218] Key type ._llcrypt registered [ 1475.087590] Key type .llcrypt registered [ 1475.990536] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 05:05:09 (1787735109) [ 1476.187163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1476.196135] alg: No test for adler32 (adler32-zlib) [ 1477.079616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1477.120922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1477.143918] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1477.723253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1477.747920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.45@tcp [ 1477.771616] LNetError: 90644:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.45@tcp to recovery [ 1477.848602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.45@tcp [ 1477.951160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1481.666841] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 05:05:15 (1787735115) [ 1481.719694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1481.731770] LNet: 91229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.736601] LNetError: 91229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.745663] LNet: Removed LNI 192.168.203.45@tcp [ 1481.764620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.797724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1481.810754] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1482.238164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1482.252542] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1483.048464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.45@tcp [ 1483.066714] LNetError: 91500:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.45@tcp to recovery [ 1483.808348] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1483.809869] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1484.224251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.45@tcp [ 1491.425370] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 05:05:24 (1787735124) [ 1492.686016] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg345-server [ 1494.312207] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 05:05:27 (1787735127) [ 1495.957113] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg345-server [ 1497.413668] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 05:05:30 (1787735130) [ 1499.042396] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg345-server [ 1500.524557] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 05:05:33 (1787735133) [ 1500.639891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1500.663207] LNet: 93123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.673520] LNetError: 93123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.679303] LNet: Removed LNI 192.168.203.45@tcp [ 1500.703232] LNet: Removed LNI 192.168.203.45@tcp1 [ 1500.721591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1500.784612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1500.801274] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1501.336810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1501.407955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1501.473198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1501.590644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1501.643742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1501.712389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1506.716086] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 05:05:39 (1787735139) [ 1506.864662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1506.907721] LNet: 93859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.922806] LNetError: 93859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1506.934022] LNetError: 93859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1506.947574] LNet: Removed LNI 192.168.203.45@tcp [ 1506.964962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1507.008285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1507.035244] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1513.626594] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 05:05:46 (1787735146) [ 1513.839522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1513.861287] LNet: 94563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.870836] LNetError: 94563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1513.885551] LNet: Removed LNI 192.168.203.45@tcp [ 1513.904809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.966977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1513.991191] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1520.485541] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 05:05:53 (1787735153) [ 1522.018159] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1523.482245] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 05:05:56 (1787735156) [ 1523.849574] LNet: 95597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1523.855903] LNetError: 95597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1523.871291] LNet: Removed LNI 192.168.203.45@tcp [ 1524.415618] Key type .llcrypt unregistered [ 1524.419184] Key type ._llcrypt unregistered [ 1529.835306] Key type ._llcrypt registered [ 1529.837964] Key type .llcrypt registered [ 1530.066641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1530.082813] alg: No test for adler32 (adler32-zlib) [ 1531.007192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1531.048417] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1538.091322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1543.090927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.145@tcp [ 1543.479873] LNet: 96022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.490746] LNetError: 96022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.506947] LNet: Removed LNI 192.168.203.45@tcp [ 1543.916689] Key type .llcrypt unregistered [ 1543.921938] Key type ._llcrypt unregistered [ 1554.896406] Key type ._llcrypt registered [ 1554.901685] Key type .llcrypt registered [ 1556.001530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1557.339426] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1558.987145] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 05:06:31 (1787735191) [ 1560.368833] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1561.770879] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 05:06:34 (1787735194) [ 1563.380678] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg345-server [ 1564.752622] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 05:06:37 (1787735197) [ 1566.214205] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg345-server [ 1567.684402] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 05:06:40 (1787735200) [ 1568.911714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1568.950499] alg: No test for adler32 (adler32-zlib) [ 1569.973800] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 1570.177375] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1571.808258] Key type lgssc registered [ 1572.824621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1588.473444] Key type lgssc unregistered [ 1588.756389] LNet: 99198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.768616] LNetError: 99198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.798026] LNet: Removed LNI 192.168.203.45@tcp [ 1589.405270] Key type .llcrypt unregistered [ 1589.409195] Key type ._llcrypt unregistered [ 1601.306731] Key type ._llcrypt registered [ 1601.308463] Key type .llcrypt registered [ 1602.388382] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 05:07:15 (1787735235) [ 1602.653655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1602.662513] alg: No test for adler32 (adler32-zlib) [ 1603.574683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.642223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1603.676992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1603.708292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1603.761840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1608.795937] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 05:07:21 (1787735241) [ 1608.972961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1608.986817] LNet: 100612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.000807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.060149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.082668] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1609.864558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.883612] LNet: 100756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.887427] LNetError: 100756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1609.900295] LNet: Removed LNI 192.168.203.45@tcp [ 1609.916494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.971908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1609.998208] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1610.471667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.488569] LNetError: 100784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.506630] LNet: Removed LNI 192.168.203.45@tcp [ 1610.532635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.607145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1610.640651] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1616.350240] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 05:07:29 (1787735249) [ 1616.496194] LNet: 101370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.513209] LNet: 101370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1616.524608] LNetError: 101370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.575730] LNet: Removed LNI 192.168.203.45@tcp [ 1617.123854] Key type .llcrypt unregistered [ 1617.127421] Key type ._llcrypt unregistered [ 1622.848896] Key type ._llcrypt registered [ 1622.850667] Key type .llcrypt registered [ 1623.002980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1623.014425] alg: No test for adler32 (adler32-zlib) [ 1623.884656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.936408] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1623.939222] LNet: *** cfs_fail_loc=e003, val=100*** [ 1624.549805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1624.567657] LNet: *** cfs_fail_loc=e003, val=100*** [ 1624.570438] LNet: Skipped 2 previous similar messages [ 1624.577682] Ping reply received with mapping: [ 1624.579277] found metadata reply NI[192.168.203.45@tcp] [ 1624.581905] 2d cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1624.584929] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1624.590819] 5a 5a 5a 5a 5a 5a 5a 5a [ 1624.796275] LNet: 101774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.809188] LNetError: 101774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1624.826507] LNet: Removed LNI 192.168.203.45@tcp [ 1625.632705] Key type .llcrypt unregistered [ 1625.637939] Key type ._llcrypt unregistered [ 1636.597337] Key type ._llcrypt registered [ 1636.599388] Key type .llcrypt registered [ 1637.825827] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 05:07:50 (1787735270) [ 1638.251304] Key type .llcrypt unregistered [ 1638.254217] Key type ._llcrypt unregistered [ 1644.785291] Key type ._llcrypt registered [ 1644.787499] Key type .llcrypt registered [ 1644.942207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1644.953637] alg: No test for adler32 (adler32-zlib) [ 1645.924972] LNetError: 102831:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1646.219457] Key type .llcrypt unregistered [ 1646.227296] Key type ._llcrypt unregistered [ 1650.383204] Key type ._llcrypt registered [ 1650.387904] Key type .llcrypt registered [ 1651.375471] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 05:08:04 (1787735284) [ 1651.972328] Key type .llcrypt unregistered [ 1651.975154] Key type ._llcrypt unregistered [ 1657.047887] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1657.050909] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1659.628358] Key type ._llcrypt registered [ 1659.633115] Key type .llcrypt registered [ 1659.833301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1659.864307] alg: No test for adler32 (adler32-zlib) [ 1660.859829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.938243] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1668.962333] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 05:08:22 (1787735302) [ 1669.194529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1669.230950] LNet: 104620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1669.237625] LNetError: 104620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1669.254871] LNet: Removed LNI 10.1.2.3@tcp [ 1669.273493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.036235] Key type .llcrypt unregistered [ 1670.043580] Key type ._llcrypt unregistered [ 1675.930992] Key type ._llcrypt registered [ 1675.937917] Key type .llcrypt registered [ 1676.190967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1676.215752] alg: No test for adler32 (adler32-zlib) [ 1677.177609] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1677.259939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1677.478937] LNet: 104912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1677.491047] LNetError: 104912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1677.515657] LNet: Removed LNI 192.168.203.45@tcp [ 1678.284522] Key type .llcrypt unregistered [ 1678.290478] Key type ._llcrypt unregistered [ 1688.568753] Key type ._llcrypt registered [ 1688.570941] Key type .llcrypt registered [ 1689.626687] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 05:08:42 (1787735322) [ 1690.480470] Key type .llcrypt unregistered [ 1690.484461] Key type ._llcrypt unregistered [ 1696.120240] Key type ._llcrypt registered [ 1696.122336] Key type .llcrypt registered [ 1696.302829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1696.326066] alg: No test for adler32 (adler32-zlib) [ 1702.528797] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 05:08:55 (1787735335) [ 1702.658685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.700043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.773755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1702.792893] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1703.364434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1703.404235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1703.453773] LNet: Removed LNI 192.168.203.45@tcp [ 1703.778464] LNet: 106668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.241459] Key type .llcrypt unregistered [ 1704.245627] Key type ._llcrypt unregistered [ 1713.492964] Key type ._llcrypt registered [ 1713.496695] Key type .llcrypt registered [ 1714.357829] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 05:09:07 (1787735347) [ 1714.727980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1714.741485] alg: No test for adler32 (adler32-zlib) [ 1715.623653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.688514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1715.713384] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1716.293535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1716.359777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.45@tcp [ 1716.398316] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1716.403668] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1716.408519] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1716, 0, 0 [ 1716.424225] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1716.428156] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1716.433647] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1716.440499] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1716, 0, 0 [ 1716.447508] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1716.451522] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1716.456659] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1716.459657] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1716, 0, 1766 [ 1716.473174] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1716.476814] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1716.481290] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1716.485210] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1716, 0, 1766 [ 1716.490826] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1721.301399] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 05:09:14 (1787735354) [ 1721.758585] LNet: 108208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1721.767927] LNetError: 108208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1721.792210] LNet: Removed LNI 192.168.203.45@tcp [ 1722.431545] Key type .llcrypt unregistered [ 1722.433746] Key type ._llcrypt unregistered [ 1727.844319] Key type ._llcrypt registered [ 1727.847578] Key type .llcrypt registered [ 1728.008844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1728.030951] alg: No test for adler32 (adler32-zlib) [ 1728.949677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1728.999282] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1735.999940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1744.230863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.276361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1744.303741] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1745.260109] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1745.415900] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1745.420309] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1745.429883] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1747.778939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1747.817869] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1757.015471] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 05:09:50 (1787735390) [ 1757.358131] LNet: 109715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.372541] LNetError: 109715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.389558] LNet: Removed LNI 192.168.203.45@tcp [ 1758.432170] LNet: Removed LNI 192.168.203.45@tcp1 [ 1758.439857] LNetError: 109715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1758.458491] LNetError: 109715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1758.484127] LNet: Removed LNI 10.1.2.3@tcp99 [ 1759.187817] Key type .llcrypt unregistered [ 1759.189689] Key type ._llcrypt unregistered [ 1765.227071] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1765.232819] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1767.671702] Key type ._llcrypt registered [ 1767.684922] Key type .llcrypt registered [ 1767.893679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1767.910811] alg: No test for adler32 (adler32-zlib) [ 1768.807302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.865544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1768.898365] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1769.516740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1769.530330] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1771.648627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1771.696843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1771.750541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1771.775127] LNetError: 110464:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1771.802225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1771.850639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1772.342359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1772.460657] LNet: 110484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1772.477377] LNetError: 110484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1772.491663] LNet: Removed LNI 10.1.2.3@tcp [ 1773.545395] LNet: Removed LNI 192.168.203.45@tcp [ 1774.142502] Key type .llcrypt unregistered [ 1774.145625] Key type ._llcrypt unregistered [ 1785.495536] Key type ._llcrypt registered [ 1785.498270] Key type .llcrypt registered [ 1786.986428] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 05:10:19 (1787735419) [ 1787.317816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1787.347457] alg: No test for adler32 (adler32-zlib) [ 1788.353856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1788.423758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1788.460164] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1789.200808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1789.285359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg345-client@tcp [ 1795.230574] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 05:10:28 (1787735428) [ 1795.376813] LNet: 112017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.388852] LNetError: 112017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.420435] LNet: Removed LNI 192.168.203.45@tcp [ 1796.048367] Key type .llcrypt unregistered [ 1796.054475] Key type ._llcrypt unregistered [ 1802.222414] Key type ._llcrypt registered [ 1802.225112] Key type .llcrypt registered [ 1802.411868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1802.421647] alg: No test for adler32 (adler32-zlib) [ 1803.310935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1803.332341] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1803.357406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1809.028873] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 05:10:41 (1787735441) [ 1810.362281] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1811.908089] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 05:10:44 (1787735444) [ 1813.310377] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1814.636297] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 05:10:47 (1787735447) [ 1814.852685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.866695] LNet: 113445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.872796] LNetError: 113445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.892948] LNet: Removed LNI 192.168.203.45@tcp [ 1814.918315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.985691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1815.017591] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1815.724810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1815.779789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1821.402863] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 05:10:54 (1787735454) [ 1821.559583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1821.574614] LNet: 114150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.579847] LNetError: 114150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.595718] LNet: Removed LNI 192.168.203.45@tcp [ 1821.618231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.673490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1821.700860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1822.400988] Key type .llcrypt unregistered [ 1822.403707] Key type ._llcrypt unregistered [ 1830.941756] Key type ._llcrypt registered [ 1830.944234] Key type .llcrypt registered [ 1831.977844] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 05:11:05 (1787735465) [ 1832.281215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1832.298259] alg: No test for adler32 (adler32-zlib) [ 1833.196471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1833.239878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1833.259439] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1833.870446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1833.933256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1838.675871] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 05:11:11 (1787735471) [ 1838.748034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1838.766312] LNet: 115708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1838.777600] LNetError: 115708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1838.797348] LNet: Removed LNI 192.168.203.45@tcp [ 1838.833081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1838.878356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1838.902514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1843.984954] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 05:11:17 (1787735477) [ 1844.111947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1844.140893] LNet: 116290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.172291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.265815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1844.606414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1844.635126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1844.702717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1845.080882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1845.125790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1845.189923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1845.215788] LNet: 116327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.256216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1845.297912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1845.704485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1845.751958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1845.807487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1846.163289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1846.195616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1851.550413] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 05:11:24 (1787735484) [ 1851.623249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1851.654129] LNet: 116921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.679982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1851.737853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1851.759456] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1852.318574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1852.336482] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 1853.019379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1853.075536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1853.296237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1853.318150] LNetError: 117206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1854.368175] LNet: Removed LNI 192.168.203.45@tcp [ 1854.370440] LNetError: 117206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1854.385570] LNet: Removed LNI 192.168.203.45@tcp2 [ 1854.401505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1854.476234] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1854.480832] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 1854.484215] LNet: 117212:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1854.935177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1855.132670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1856.160829] LNet: Removed LNI 192.168.203.45@tcp [ 1856.172176] LNet: Removed LNI 192.168.203.45@tcp2 [ 1856.202487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.267544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1856.306435] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1857.369644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1857.393692] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 1858.220259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1858.276240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1858.357730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1858.563911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1858.583515] LNet: 117534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.587962] LNet: 117534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1858.593578] LNetError: 117534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1858.598492] LNetError: 117534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1858.604810] LNet: Removed LNI 192.168.203.45@tcp [ 1859.636612] LNet: Removed LNI 192.168.203.45@tcp2 [ 1859.657468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1859.721808] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1859.726240] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 1859.731782] LNet: 117540:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1860.155432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1860.315968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1860.335963] LNet: Removed LNI 192.168.203.45@tcp [ 1861.358222] LNet: Removed LNI 192.168.203.45@tcp2 [ 1861.401674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1861.468429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1861.487089] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1862.071595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1862.083127] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 1862.687440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1862.757863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1862.925369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1862.947212] LNetError: 117856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1862.951182] LNetError: 117856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1862.955105] LNet: Removed LNI 192.168.203.45@tcp [ 1862.963672] LNet: Removed LNI 192.168.203.45@tcp2 [ 1862.987562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1863.075179] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1863.087808] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 1863.094175] LNet: 117862:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1863.433339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1863.593602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1864.672130] LNet: Removed LNI 192.168.203.45@tcp [ 1864.688544] LNet: Removed LNI 192.168.203.45@tcp2 [ 1864.718693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1864.784798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1864.807826] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1865.447672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1865.465702] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 1866.373711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1866.407582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1866.465586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1866.668357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1866.689811] LNet: 118184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.698977] LNet: 118184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1867.744895] LNet: Removed LNI 192.168.203.45@tcp [ 1868.773929] LNet: Removed LNI 192.168.203.45@tcp2 [ 1868.797950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1868.895394] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1868.900856] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 1868.905476] LNet: 118190:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1869.525303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1869.817539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1869.850965] LNet: Removed LNI 192.168.203.45@tcp [ 1870.888879] LNet: Removed LNI 192.168.203.45@tcp2 [ 1870.915563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1870.975343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1870.991974] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1871.625838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1871.649755] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 1872.470928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1872.507752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1872.572141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1873.064127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1873.083992] LNetError: 118524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1873.088356] LNetError: 118524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1873.095631] LNet: Removed LNI 192.168.203.45@tcp [ 1873.111234] LNet: Removed LNI 192.168.203.45@tcp2 [ 1873.144403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1873.197827] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1873.204710] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 1873.213752] LNet: 118530:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1873.852503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1874.216129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1875.297729] LNet: Removed LNI 192.168.203.45@tcp [ 1875.323455] LNet: Removed LNI 192.168.203.45@tcp2 [ 1875.374602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1875.537083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1875.631831] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1876.613131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1876.643655] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 1877.622800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1877.673938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1877.739379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1877.804052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1878.195718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1879.264131] LNet: Removed LNI 192.168.203.45@tcp [ 1880.295057] LNet: Removed LNI 192.168.203.45@tcp2 [ 1880.317109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.415860] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1880.421697] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 1881.214285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1886.468352] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1887.477923] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 05:12:00 (1787735520) [ 1887.680674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1887.693630] LNet: 119649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1887.696468] LNet: 119649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1887.700179] LNet: Removed LNI 192.168.203.45@tcp [ 1888.744465] LNet: Removed LNI 192.168.203.45@tcp2 [ 1888.767572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1889.061522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1889.088158] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1889.686501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1889.721570] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1890.423251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1890.440863] LNetError: 119922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1890.447378] LNetError: 119922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1890.454027] LNet: Removed LNI 10.1.2.3@tcp [ 1890.463769] LNet: Removed LNI 192.168.203.45@tcp [ 1890.493544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1890.573624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1890.616779] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1890.622683] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1890.968293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1890.985319] LNet: Removed LNI 192.168.203.45@tcp [ 1891.001224] LNet: Removed LNI 10.1.2.3@tcp [ 1896.074788] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 05:12:09 (1787735529) [ 1896.331312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1896.367079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1896.411137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1896.424499] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1896.972868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1897.010268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1897.050502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1897.095317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1897.117470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1897.143667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1897.176309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1897.194363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1897.213214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1897.239325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1897.260761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1897.405159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1903.371760] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 05:12:16 (1787735536) [ 1903.466950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1904.551884] LNet: Removed LNI 192.168.203.45@tcp [ 1904.619393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1904.763754] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1904.987388] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1904.996244] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1905.008152] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1907.444041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1907.513275] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1908.331433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1908.348391] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1910.651449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1910.701569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1910.767329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.45@tcp -m PUT -e local_timeout -r 1 [ 1910.850605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1910.913185] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1910.918868] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1911.120500] LNetError: 121327:0:(lib-move.c:4801:lnet_parse()) 192.168.203.45@tcp, src 192.168.203.45@tcp: Dropping PUT (error -108 looking up sender) [ 1912.161115] LNet: Removed LNI 10.1.2.3@tcp [ 1913.193613] LNet: Removed LNI 192.168.203.45@tcp [ 1913.741243] Key type .llcrypt unregistered [ 1913.746531] Key type ._llcrypt unregistered [ 1924.605461] Key type ._llcrypt registered [ 1924.606913] Key type .llcrypt registered [ 1925.644264] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 05:12:38 (1787735558) [ 1925.987712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1926.003545] alg: No test for adler32 (adler32-zlib) [ 1926.891451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1927.198094] LNet: 122641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1927.690389] Key type .llcrypt unregistered [ 1927.696864] Key type ._llcrypt unregistered [ 1932.429993] Key type ._llcrypt registered [ 1932.432309] Key type .llcrypt registered [ 1932.573274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1932.586259] alg: No test for adler32 (adler32-zlib) [ 1933.481531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1933.502832] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1939.530340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1939.678909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1940.141399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.145@tcp [ 1940.387242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1940.405623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1940.439597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1940.477348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1940.515772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1940.558694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1940.588852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1940.613470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1940.634559] LNet: 123094:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1940.662759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1940.692187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1940.705332] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.145@tcp [ 1940.721810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1940.749519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1940.789279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1940.826071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1941.419776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1941.618405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1941.669694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1941.699799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1941.750093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1941.818380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1941.869862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1941.931417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1941.980904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1942.005742] LNet: 123211:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1942.035670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1942.066914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1942.104465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1942.136442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1942.170304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1942.208613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1942.788255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.145@tcp [ 1942.962639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1942.989023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1943.011118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1943.036761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1943.065985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1943.095275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1943.117892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1943.142617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1943.156470] LNet: 123329:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1943.174803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1943.205208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1943.232563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1943.267670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1943.318679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1943.360198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1943.865568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1944.070442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1944.101191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1944.122799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1944.147519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1944.174457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1944.198840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1944.222559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1944.245291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1944.266595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1944.290741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1944.341357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1944.384808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1944.418750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1944.467915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1944.836604] LNet: 123496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.849173] LNetError: 123496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.869672] LNet: Removed LNI 192.168.203.45@tcp [ 1945.271762] Key type .llcrypt unregistered [ 1945.277284] Key type ._llcrypt unregistered [ 1957.284982] Key type ._llcrypt registered [ 1957.287095] Key type .llcrypt registered [ 1958.351964] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 05:13:11 (1787735591) [ 1958.622921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1958.637341] alg: No test for adler32 (adler32-zlib) [ 1959.500928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1959.937335] LNet: 124382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.415298] Key type .llcrypt unregistered [ 1960.418488] Key type ._llcrypt unregistered [ 1965.535943] Key type ._llcrypt registered [ 1965.538198] Key type .llcrypt registered [ 1965.731542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1965.742483] alg: No test for adler32 (adler32-zlib) [ 1966.640615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1966.675151] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1972.491623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1979.326039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1979.356719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1979.377522] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1980.064057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1980.491507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.145@tcp [ 1980.692325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1980.716667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1980.756918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1980.792082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1980.816804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1980.843608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1980.885563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1980.919670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1980.939445] LNet: 125037:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1980.964230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1981.008577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1981.024655] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.145@tcp [ 1981.043690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1981.076214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1981.119510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1981.153485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1981.713511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1981.926957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1981.957586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1981.986049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1982.009993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1982.029712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1982.051569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1982.082162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1982.106943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1982.138910] LNet: 125154:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1982.161423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1982.197317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1982.222792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1982.248953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1982.289433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1982.324824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1982.893177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.145@tcp [ 1983.094905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1983.117357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1983.164824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1983.188334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1983.210779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1983.253186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1983.283904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1983.312333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1983.333265] LNet: 125271:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1983.353541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1983.388573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1983.430801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1983.463778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1983.498556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1983.534464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1984.094487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1984.275490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1984.301898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1984.323644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1984.345840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1984.371904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1984.393657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1984.417663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1984.443672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1984.466584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1984.497167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1984.526767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1984.556506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1984.609949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1984.638628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1988.855390] LNet: 125473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1988.862054] LNetError: 125473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1988.873775] LNet: Removed LNI 192.168.203.45@tcp [ 1988.885617] LNet: Removed LNI 192.168.203.45@tcp1 [ 1989.221851] Key type .llcrypt unregistered [ 1989.225345] Key type ._llcrypt unregistered [ 2001.404168] Key type ._llcrypt registered [ 2001.405792] Key type .llcrypt registered [ 2002.496747] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 05:13:55 (1787735635) [ 2002.782074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2002.798388] alg: No test for adler32 (adler32-zlib) [ 2003.689031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2003.732990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2003.751115] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2004.239440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2004.252535] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2004.931386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2005.003951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2006.048252] LNet: Removed LNI 192.168.203.45@tcp [ 2010.857700] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 05:14:04 (1787735644) [ 2012.000828] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2013.249101] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 05:14:06 (1787735646) [ 2018.107911] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 05:14:11 (1787735651) [ 2022.919598] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 05:14:16 (1787735656) [ 2028.170762] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 05:14:21 (1787735661) [ 2033.948688] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 05:14:27 (1787735667) [ 2038.808644] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 05:14:32 (1787735672) [ 2044.340142] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 05:14:37 (1787735677) [ 2049.445922] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 05:14:42 (1787735682) [ 2054.685446] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 05:14:48 (1787735688) [ 2060.078429] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 05:14:53 (1787735693) [ 2066.445413] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 05:14:59 (1787735699) [ 2072.963238] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 05:15:06 (1787735706) [ 2078.608280] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 05:15:11 (1787735711) [ 2083.348136] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 05:15:16 (1787735716) [ 2088.438070] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 05:15:21 (1787735721) [ 2093.965532] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 05:15:27 (1787735727) [ 2100.069794] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 05:15:33 (1787735733) [ 2105.778779] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 05:15:38 (1787735738) [ 2105.880197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2105.908372] LNet: 137158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.917503] LNetError: 137158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2105.947367] LNet: Removed LNI 192.168.203.45@tcp2 [ 2105.999524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.140353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2106.172361] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2107.318409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2107.346450] LNet: 137306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2107.354233] LNetError: 137306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2107.369741] LNet: Removed LNI 192.168.203.45@tcp [ 2107.398547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2107.462417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2107.488069] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2112.956208] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 05:15:46 (1787735746) [ 2118.298209] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 05:15:51 (1787735751) [ 2123.564866] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 05:15:56 (1787735756) [ 2129.060567] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 05:16:02 (1787735762) [ 2134.332601] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 05:16:07 (1787735767) [ 2139.856841] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 05:16:13 (1787735773) [ 2146.358921] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 05:16:19 (1787735779) [ 2152.501377] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 05:16:25 (1787735785) [ 2158.048541] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 05:16:31 (1787735791) [ 2163.481660] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 05:16:36 (1787735796) [ 2168.356797] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 05:16:41 (1787735801) [ 2173.585976] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 05:16:46 (1787735806) [ 2179.394489] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1986 sec ========== 05:16:52 (1787735812) [ 2179.750040] LNet: 144770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2179.753800] LNetError: 144770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2179.768173] LNet: Removed LNI 192.168.203.45@tcp [ 2180.421924] Key type .llcrypt unregistered [ 2180.425461] Key type ._llcrypt unregistered [ 2193.035521] Key type ._llcrypt registered [ 2193.038771] Key type .llcrypt registered [ 2193.205657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2193.234567] alg: No test for adler32 (adler32-zlib) [ 2194.259734] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 2194.471690] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2196.168174] Key type lgssc registered [ 2197.009531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2247.638837] Lustre: Mounted lustre-client - version 2.17.57_80_g6de652b [ 2251.126540] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2258.725650] Lustre: Unmounted lustre-client [ 2294.844814] Key type lgssc unregistered [ 2295.054536] LNet: 147727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2295.061301] LNetError: 147727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2295.084371] LNet: Removed LNI 192.168.203.45@tcp [ 2295.646557] Key type .llcrypt unregistered [ 2295.650198] Key type ._llcrypt unregistered