[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 432621107 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002362] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006984] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009130] LSM: Security Framework initializing [ 0.010039] Yama: becoming mindful. [ 0.012010] SELinux: Initializing. [ 0.013015] *** VALIDATE selinux *** [ 0.020795] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024721] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026095] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029066] *** VALIDATE tmpfs *** [ 0.030417] *** VALIDATE proc *** [ 0.031242] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.033264] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036029] Spectre V2 : User space: Vulnerable [ 0.037010] Speculative Store Bypass: Vulnerable [ 0.039412] debug: unmapping init [mem 0xffffffffac659000-0xffffffffac660fff] [ 0.042000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042792] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043025] ... version: 2 [ 0.044013] ... bit width: 48 [ 0.045011] ... generic registers: 4 [ 0.046011] ... value mask: 0000ffffffffffff [ 0.047012] ... max period: 00007fffffffffff [ 0.048017] ... fixed-purpose events: 3 [ 0.049072] ... event mask: 000000070000000f [ 0.050315] rcu: Hierarchical SRCU implementation. [ 0.052519] smp: Bringing up secondary CPUs ... [ 0.053613] x86: Booting SMP configuration: [ 0.054019] .... node #0, CPUs: #1 #2 #3 [ 0.058165] smp: Brought up 1 node, 4 CPUs [ 0.060018] smpboot: Max logical packages: 1 [ 0.061013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.131000] node 0 deferred pages initialised in 69ms [ 0.135011] devtmpfs: initialized [ 0.136111] x86/mm: Memory block size: 128MB [ 0.138843] gcov: version magic: 0x41383552 [ 0.140333] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141064] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142223] pinctrl core: initialized pinctrl subsystem [ 0.143187] [ 0.143770] ************************************************************* [ 0.144014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145013] ** ** [ 0.146013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.147016] ** ** [ 0.148014] ** This means that this kernel is built to expose internal ** [ 0.149016] ** IOMMU data structures, which may compromise security on ** [ 0.150014] ** your system. ** [ 0.151013] ** ** [ 0.152012] ** If you see this message and you are not debugging the ** [ 0.153015] ** kernel, report this immediately to your vendor! ** [ 0.154016] ** ** [ 0.155012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156072] ************************************************************* [ 0.157760] NET: Registered protocol family 16 [ 0.158448] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.159070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.160062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.161482] cpuidle: using governor menu [ 0.163738] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.164522] PCI: Using configuration type 1 for base access [ 0.165200] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.172143] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.175074] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.178128] cryptd: max_cpu_qlen set to 1000 [ 0.181264] ACPI: Added _OSI(Module Device) [ 0.183013] ACPI: Added _OSI(Processor Device) [ 0.184011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.186012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.191363] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.197531] ACPI: Interpreter enabled [ 0.198045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.199011] ACPI: Using IOAPIC for interrupt routing [ 0.200103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.201327] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.211112] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.212038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.213020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.214076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.216766] acpiphp: Slot [2] registered [ 0.217095] acpiphp: Slot [5] registered [ 0.218089] acpiphp: Slot [6] registered [ 0.219083] acpiphp: Slot [3] registered [ 0.219871] acpiphp: Slot [4] registered [ 0.220073] acpiphp: Slot [7] registered [ 0.221145] acpiphp: Slot [8] registered [ 0.222135] acpiphp: Slot [9] registered [ 0.222991] acpiphp: Slot [10] registered [ 0.223061] acpiphp: Slot [11] registered [ 0.223914] acpiphp: Slot [12] registered [ 0.224057] acpiphp: Slot [13] registered [ 0.225019] acpiphp: Slot [14] registered [ 0.226028] acpiphp: Slot [15] registered [ 0.226991] acpiphp: Slot [16] registered [ 0.227068] acpiphp: Slot [17] registered [ 0.227952] acpiphp: Slot [18] registered [ 0.228058] acpiphp: Slot [19] registered [ 0.229103] acpiphp: Slot [20] registered [ 0.230043] acpiphp: Slot [21] registered [ 0.231092] acpiphp: Slot [22] registered [ 0.232072] acpiphp: Slot [23] registered [ 0.233094] acpiphp: Slot [24] registered [ 0.235103] acpiphp: Slot [25] registered [ 0.236062] acpiphp: Slot [26] registered [ 0.237086] acpiphp: Slot [27] registered [ 0.238062] acpiphp: Slot [28] registered [ 0.239104] acpiphp: Slot [29] registered [ 0.240077] acpiphp: Slot [30] registered [ 0.241103] acpiphp: Slot [31] registered [ 0.241764] PCI host bridge to bus 0000:00 [ 0.243017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.244076] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.246018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.249021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.252020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.255057] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.257192] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.261305] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.265364] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.274011] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.278135] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.281018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.283015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.286016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.288523] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.291821] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.295054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.298636] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.304013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.314870] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.319013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.325553] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.331016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.337017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.348015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.360126] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.364015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.367012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.376017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.382673] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.384225] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.386237] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.387306] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.389136] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.393047] iommu: Default domain type: Passthrough [ 0.394485] SCSI subsystem initialized [ 0.395145] ACPI: bus type USB registered [ 0.397116] usbcore: registered new interface driver usbfs [ 0.400092] usbcore: registered new interface driver hub [ 0.402082] usbcore: registered new device driver usb [ 0.404144] pps_core: LinuxPPS API ver. 1 registered [ 0.406009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.409062] PTP clock support registered [ 0.411124] EDAC MC: Ver: 3.0.0 [ 0.413115] PCI: Using ACPI for IRQ routing [ 0.414585] NetLabel: Initializing [ 0.416013] NetLabel: domain hash size = 128 [ 0.417000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.417000] NetLabel: unlabeled traffic allowed by default [ 0.419117] vgaarb: loaded [ 0.420254] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.422013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.431255] clocksource: Switched to clocksource kvm-clock [ 0.530546] VFS: Disk quotas dquot_6.6.0 [ 0.532341] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.535312] *** VALIDATE ramfs *** [ 0.537116] *** VALIDATE hugetlbfs *** [ 0.539133] pnp: PnP ACPI init [ 0.543385] pnp: PnP ACPI: found 6 devices [ 0.559873] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.564427] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.566428] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.568264] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.570466] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.572569] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.574723] NET: Registered protocol family 2 [ 0.576409] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.580847] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.584286] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.588748] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.591349] TCP: Hash tables configured (established 65536 bind 65536) [ 0.594273] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.597348] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.599383] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.601666] NET: Registered protocol family 1 [ 0.603449] RPC: Registered named UNIX socket transport module. [ 0.605387] RPC: Registered udp transport module. [ 0.606775] RPC: Registered tcp transport module. [ 0.608505] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.610918] NET: Registered protocol family 44 [ 0.612405] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.614199] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.616427] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.618188] PCI: CLS 0 bytes, default 64 [ 0.619882] Unpacking initramfs... [ 1.902043] debug: unmapping init [mem 0xffff9cd13cc64000-0xffff9cd13ffcffff] [ 1.904750] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.906258] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.908580] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.414414] Initialise system trusted keyrings [ 2.416940] Key type blacklist registered [ 2.418984] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.429236] zbud: loaded [ 2.431627] *** VALIDATE nfs *** [ 2.432410] *** VALIDATE nfs4 *** [ 2.433504] pstore: using deflate compression [ 2.435649] Platform Keyring initialized [ 2.530321] NET: Registered protocol family 38 [ 2.532224] Key type asymmetric registered [ 2.533775] Asymmetric key parser 'x509' registered [ 2.535211] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.538450] io scheduler mq-deadline registered [ 2.540397] io scheduler kyber registered [ 2.542254] io scheduler bfq registered [ 2.544159] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.547342] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.549937] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.553355] ACPI: Power Button [PWRF] [ 2.557893] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.563881] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.574132] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.602740] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.631988] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.638647] Non-volatile memory driver v1.3 [ 2.640691] Linux agpgart interface v0.103 [ 2.671231] virtio_blk virtio1: [vda] 146936 512-byte logical blocks (75.2 MB/71.7 MiB) [ 2.674398] vda: detected capacity change from 0 to 75231232 [ 2.688291] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.691277] vdb: detected capacity change from 0 to 1073741824 [ 2.696437] libphy: Fixed MDIO Bus: probed [ 2.702742] usbcore: registered new interface driver usbserial_generic [ 2.704132] usbserial: USB Serial support registered for generic [ 2.705360] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.707995] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.709652] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.711970] mousedev: PS/2 mouse device common for all mice [ 2.714728] rtc_cmos 00:05: RTC can wake from S4 [ 2.717782] rtc_cmos 00:05: registered as rtc0 [ 2.717978] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.719730] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.724996] intel_pstate: CPU model not supported [ 2.728280] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.728843] hid: raw HID events driver (C) Jiri Kosina [ 2.734587] usbcore: registered new interface driver usbhid [ 2.737253] usbhid: USB HID core driver [ 2.739345] drop_monitor: Initializing network drop monitor service [ 2.742390] Initializing XFRM netlink socket [ 2.744572] NET: Registered protocol family 10 [ 2.748221] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.748507] Segment Routing with IPv6 [ 2.753404] NET: Registered protocol family 17 [ 2.755048] mpls_gso: MPLS GSO support [ 2.760820] RAS: Correctable Errors collector initialized. [ 2.763700] AVX version of gcm_enc/dec engaged. [ 2.765388] AES CTR mode by8 optimization enabled [ 2.862946] sched_clock: Marking stable (2862895960, 0)->(3780005798, -917109838) [ 2.866722] registered taskstats version 1 [ 2.868901] Loading compiled-in X.509 certificates [ 2.871822] zswap: loaded using pool lzo/zbud [ 2.898748] Key type big_key registered [ 2.910410] Key type encrypted registered [ 2.912832] ima: No TPM chip found, activating TPM-bypass! [ 2.915204] ima: Allocated hash algorithm: sha1 [ 2.917099] ima: No architecture policies found [ 2.918984] evm: Initialising EVM extended attributes: [ 2.921069] evm: security.selinux [ 2.922428] evm: security.ima [ 2.923779] evm: security.capability [ 2.925263] evm: HMAC attrs: 0x1 [ 2.927785] rtc_cmos 00:05: setting system clock to 2026-08-27 14:39:17 UTC (1787841557) [ 2.936938] debug: unmapping init [mem 0xffffffffad603000-0xffffffffad7fffff] [ 2.940363] debug: unmapping init [mem 0xffffffffac382000-0xffffffffac658fff] [ 2.951114] Write protecting the kernel read-only data: 28672k [ 2.955718] debug: unmapping init [mem 0xffffffffaaa03000-0xffffffffaabfffff] [ 2.958693] debug: unmapping init [mem 0xffffffffab314000-0xffffffffab3fffff] [ 2.988533] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.997325] systemd[1]: Detected virtualization kvm. [ 2.999335] systemd[1]: Detected architecture x86-64. [ 3.001694] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.032197] systemd[1]: No hostname configured. [ 3.033488] systemd[1]: Set hostname to . [ 3.035214] random: systemd: uninitialized urandom read (16 bytes read) [ 3.037963] systemd[1]: Initializing machine ID from random generator. [ 3.100767] random: ln: uninitialized urandom read (6 bytes read) [ 3.182605] random: systemd: uninitialized urandom read (16 bytes read) [ 3.184509] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.188725] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.192612] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.807638] device-mapper: uevent: version 1.0.3 [ 3.810200] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.855275] virtio_net virtio0 ens2: renamed from eth0 [ 4.909913] scsi host0: ata_piix [ 4.953333] scsi host1: ata_piix [ 4.955200] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.957963] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 6.384098] random: fast init done [ 8.765507] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.430708] random: crng init done [ 9.432280] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.988788] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.223957] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.451807] SELinux: Disabled at runtime. [ 11.507568] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.516821] systemd[1]: Detected virtualization kvm. [ 11.518825] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.206226] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.209531] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.214693] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.218602] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.222214] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.234172] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.254812] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [[ 12.437996] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.861920] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.311077] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.362515] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.379794] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.409304] EDAC sbridge: Ver: 1.1.2 [ 14.530672] Key type dns_resolver registered [ 14.849198] NFS: Registering the id_resolver key type [ 14.851359] Key type id_resolver registered [ 14.853477] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg210-client login: [ 42.848021] hrtimer: interrupt took 3026039 ns [ 67.356514] libcfs: loading out-of-tree module taints kernel. [ 67.802256] Key type ._llcrypt registered [ 67.810186] Key type .llcrypt registered [ 68.669528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 68.686677] alg: No test for adler32 (adler32-zlib) [ 70.262170] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 71.188685] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 72.824595] Key type lgssc registered [ 75.791674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 211.291503] Lustre: Mounted lustre-client - version 2.17.57_92_gffc8991 [ 216.780934] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 229.582887] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing check_logdir /tmp/testlogs/ [ 235.172415] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing yml_node [ 237.027638] Lustre: lustre-OST0000-osc-ffff9cd188251000: disconnect after 23s idle [ 239.857213] Lustre: DEBUG MARKER: Client: 2.17.57.92 [ 242.197872] Lustre: DEBUG MARKER: MDS: 2.17.57.92 [ 244.874890] Lustre: DEBUG MARKER: OSS: 2.17.57.92 [ 246.365399] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Aug 27 10:43:19 EDT 2026 [ 264.272402] Lustre: Unmounted lustre-client [ 294.518876] Key type lgssc unregistered [ 294.931665] LNet: 8411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 294.954311] LNetError: 8411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 294.967305] LNet: Removed LNI 192.168.202.10@tcp [ 295.883174] Key type .llcrypt unregistered [ 295.885144] Key type ._llcrypt unregistered [ 324.041307] Key type ._llcrypt registered [ 324.045911] Key type .llcrypt registered [ 324.328635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 324.360319] alg: No test for adler32 (adler32-zlib) [ 325.325379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 325.449095] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 325.477914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 329.946846] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 330.332777] LNet: 9315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.346744] LNetError: 9315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 330.381232] LNet: Removed LNI 192.168.202.10@tcp [ 331.051604] Key type .llcrypt unregistered [ 331.056584] Key type ._llcrypt unregistered [ 337.423244] Key type ._llcrypt registered [ 337.426609] Key type .llcrypt registered [ 338.639777] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:44:51 (1787841891) [ 338.967595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 338.979882] alg: No test for adler32 (adler32-zlib) [ 339.844237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.902975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.919282] LNet: 9733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.933459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.967735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 339.980900] LNet: 9740:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 349.001922] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:45:02 (1787841902) [ 349.261194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.286978] LNet: 10318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.318672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.412807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 359.662611] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:45:11 (1787841911) [ 359.804134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.848218] LNet: 10899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.907879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.076300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 368.682142] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:45:22 (1787841922) [ 369.523821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.553086] LNet: 11489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.579364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.644180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 378.362345] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:45:31 (1787841931) [ 378.500972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.527084] LNet: 12072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.608340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.690346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 379.337257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 386.353697] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:45:39 (1787841939) [ 387.053481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.113259] LNet: 12667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.162295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.288809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 388.406209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 388.845443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 397.285628] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:45:50 (1787841950) [ 398.050786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.110835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.229125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 405.401172] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:45:58 (1787841958) [ 405.571796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.645188] LNet: 13848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 405.656509] LNet: 13848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 405.700405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.806152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 405.857569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 406.435643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 406.481888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 406.812269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 406.858712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 407.413326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 407.450854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 407.921210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 407.955946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 408.485832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 408.521806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 409.037849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 409.072543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 409.483434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 409.540977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 410.173708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 410.215220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 410.681564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 410.730403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 411.363471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 411.408776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 412.092147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 412.143901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 412.955885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 413.039784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 420.711530] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:46:13 (1787841973) [ 420.947741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.033402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.137173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 421.215833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 429.077352] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:46:22 (1787841982) [ 429.312931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.397628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.513978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 429.595740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 437.080983] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:46:30 (1787841990) [ 437.223633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.296437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.434967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 437.467705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 444.009518] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:46:37 (1787841997) [ 444.304402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.335173] LNet: 16290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 444.341595] LNet: 16290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 444.366627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.463391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 444.528836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 452.097737] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:46:45 (1787842005) [ 452.256988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.332913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.477753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 452.543448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 459.876431] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:46:53 (1787842013) [ 460.234543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.301570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.412288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 460.451917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 466.439562] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:46:59 (1787842019) [ 466.582970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.655455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.801388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 467.048548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 472.998195] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:47:06 (1787842026) [ 473.136807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.184356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.378544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 473.634753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 480.541943] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:47:13 (1787842033) [ 480.781704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.843630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.943457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 481.119549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 488.147787] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:47:21 (1787842041) [ 488.338253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.412236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.109724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 489.615703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 496.448194] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:47:29 (1787842049) [ 496.757317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.838799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.932868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 496.975335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 504.517894] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:47:37 (1787842057) [ 504.873747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.969226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.043512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 505.078252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 512.582972] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:47:45 (1787842065) [ 512.773919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.846931] LNet: 21727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.860696] LNet: 21727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 512.932952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.039363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 520.381984] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:47:53 (1787842073) [ 520.665964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.761329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.866931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 528.289838] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:48:01 (1787842081) [ 528.420018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.470957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.603431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 529.353969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 537.082836] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:48:10 (1787842090) [ 537.454958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.526644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.660819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 537.731979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 545.349132] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:48:18 (1787842098) [ 545.465764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.531751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.672863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 545.709160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 552.780109] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:48:25 (1787842105) [ 552.899043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.968657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.126244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 553.173965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 559.644901] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:48:33 (1787842113) [ 559.870801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.939134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.017193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 560.062592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 566.157689] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:48:39 (1787842119) [ 566.338648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.364822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.413773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 566.491164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 566.526266] LNetError: 25846:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 566.580457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 566.622664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 574.195639] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:48:47 (1787842127) [ 574.334473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.411524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.526247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.9PAvmIXc5i [ 574.606074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.9PAvmIXc5i [ 574.670478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.9PAvmIXc5i [ 581.591087] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:48:54 (1787842134) [ 581.673269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.721767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.778281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 581.810509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 581.847244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 588.871128] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 10:49:01 (1787842141) [ 589.204192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.285969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.375992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 589.428697] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 591.076837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 592.571182] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 594.475067] Key type lgssc registered [ 595.973478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 600.033808] LNetError: 27607:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.10@tcp99 has no available nets [ 603.061724] Key type lgssc unregistered [ 603.374261] LNetError: 28716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 603.383661] LNet: Removed LNI 192.168.202.10@tcp [ 603.822489] Key type .llcrypt unregistered [ 603.823590] Key type ._llcrypt unregistered [ 610.612107] Key type ._llcrypt registered [ 610.616269] Key type .llcrypt registered [ 611.867422] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 10:49:25 (1787842165) [ 612.130418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 612.141671] alg: No test for adler32 (adler32-zlib) [ 613.064746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.159733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 616.091513] LNet: Message forwarding will be enabled in 2s [ 619.168125] LNet: Message forwarding enabled [ 619.638143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 619.656166] LNet: Message forwarding will stop in 2s to 6s [ 624.221659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 624.966304] LNet: Message forwarding disabled [ 626.817827] LNet: Message forwarding will be enabled in 2s [ 630.112368] LNet: Message forwarding enabled [ 633.368026] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 10:49:46 (1787842186) [ 633.469372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.492819] LNet: 30132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.954619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.074710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 640.966903] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:49:54 (1787842194) [ 641.092754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.139323] LNet: 30716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.181175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.281977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 641.312746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 641.343842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 641.377158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 641.419331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 641.464355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 641.528822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 641.567372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 641.620995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 641.715650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 641.759625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 641.805660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 641.838467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 641.872243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 641.917706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 641.952948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 641.984643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 642.014834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 642.050951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 642.071967] LNetError: 30763:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 647.886734] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:50:01 (1787842201) [ 647.995827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.021800] LNet: 31329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 648.086647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.189369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 654.469692] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:50:07 (1787842207) [ 656.571832] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg210-server [ 658.219749] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:50:11 (1787842211) [ 658.576597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.604628] LNet: 32266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.636241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.724653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 658.768853] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 659.811988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[11-15]@tcp [ 659.831636] LNet: 32399:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 659.881887] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.11@tcp is being used as a gateway but routing feature is not turned on [ 659.906236] LNetError: 32270:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.11@tcp has gone from up to down [ 660.902596] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.15@tcp is being used as a gateway but routing feature is not turned on [ 660.910986] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 661.926470] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.14@tcp is being used as a gateway but routing feature is not turned on [ 661.941592] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 663.977831] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.15@tcp is being used as a gateway but routing feature is not turned on [ 664.001036] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 666.886562] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:50:20 (1787842220) [ 668.072114] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.15@tcp is being used as a gateway but routing feature is not turned on [ 668.089881] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 668.989322] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg210-server [ 670.851540] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:50:24 (1787842224) [ 671.011719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.045258] LNet: 33335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.059864] LNetError: 33335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.106403] LNet: Removed LNI 192.168.202.10@tcp [ 671.139519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.203488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 671.237197] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 671.798195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[11-16/2]@tcp [ 671.830521] LNet: 33453:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 671.837632] LNet: 33453:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 671.866668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[11-16/2]@tcp [ 671.868066] LNetError: 33340:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.15@tcp has gone from up to down [ 671.908318] LNetError: 33340:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 678.515641] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:50:31 (1787842231) [ 678.637864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.679496] LNetError: 34040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.713309] LNet: Removed LNI 192.168.202.10@tcp [ 678.748158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.942578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 679.031880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.052756] LNet: 34060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.065775] LNet: 34060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 679.130326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.208891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 679.298937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.334075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.414407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 679.508485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.548821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.602902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 679.652885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.692530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.755602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 679.873588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.958562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.066857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 680.138835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.175161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.229994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 680.355574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.424171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.478130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 680.541227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.568706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.603736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 680.661599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.691177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.752512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 680.840718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.892882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.951730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 681.000573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 681.039434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.126268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 688.566966] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:50:41 (1787842241) [ 690.697918] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg210-server [ 692.989206] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:50:45 (1787842245) [ 694.770724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg210-server [ 696.869190] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:50:50 (1787842250) [ 697.267704] LNet: 35447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.272354] LNet: 35447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 698.006262] Key type .llcrypt unregistered [ 698.011912] Key type ._llcrypt unregistered [ 704.181792] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 704.366587] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 704.379130] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 704.393245] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 707.065359] Key type ._llcrypt registered [ 707.067695] Key type .llcrypt registered [ 707.278345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 707.292371] alg: No test for adler32 (adler32-zlib) [ 708.299457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.377750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 708.459694] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 709.488518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 709.559602] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 711.938204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 711.978368] LNet: Removed LNI 10.1.2.3@tcp [ 712.175848] LNet: 36175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.185458] LNetError: 36175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 712.201189] LNet: Removed LNI 192.168.202.10@tcp [ 713.131656] Key type .llcrypt unregistered [ 713.141697] Key type ._llcrypt unregistered [ 726.420772] Key type ._llcrypt registered [ 726.422904] Key type .llcrypt registered [ 727.859638] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:51:20 (1787842280) [ 728.348876] Key type .llcrypt unregistered [ 728.350716] Key type ._llcrypt unregistered [ 734.192061] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 734.253296] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 734.258506] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 734.268453] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 735.866747] Key type ._llcrypt registered [ 735.869671] Key type .llcrypt registered [ 736.051428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 736.069790] alg: No test for adler32 (adler32-zlib) [ 736.946709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.999512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 737.023192] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 737.650844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 737.667735] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 740.045810] LNet: 37671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.052476] LNetError: 37671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 740.057511] LNet: Removed LNI 10.1.2.3@tcp [ 740.069040] LNet: Removed LNI 192.168.202.10@tcp [ 740.586800] Key type .llcrypt unregistered [ 740.590932] Key type ._llcrypt unregistered [ 751.990465] Key type ._llcrypt registered [ 751.993810] Key type .llcrypt registered [ 753.271824] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:51:46 (1787842306) [ 753.853917] Key type .llcrypt unregistered [ 753.857100] Key type ._llcrypt unregistered [ 759.547307] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 759.652045] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 759.661676] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 759.672978] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 762.537974] Key type ._llcrypt registered [ 762.539860] Key type .llcrypt registered [ 762.759050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 762.794519] alg: No test for adler32 (adler32-zlib) [ 763.724874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.791160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 763.847038] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 765.784066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 765.828848] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 767.915875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 767.951131] LNet: Removed LNI 10.1.2.3@tcp [ 767.970486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 768.035074] LNet: Removed LNI 10.1.2.31@tcp [ 768.066318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 768.100205] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 768.791269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 768.808525] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 770.056901] LNet: 39598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.065856] LNetError: 39598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 770.075199] LNet: Removed LNI 10.1.2.31@tcp [ 770.083191] LNet: Removed LNI 10.1.2.3@tcp [ 770.751462] Key type .llcrypt unregistered [ 770.754295] Key type ._llcrypt unregistered [ 782.263628] Key type ._llcrypt registered [ 782.265773] Key type .llcrypt registered [ 783.277553] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 10:52:16 (1787842336) [ 783.670511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 783.694027] alg: No test for adler32 (adler32-zlib) [ 784.578759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.631955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 784.661484] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 785.661324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.686876] LNet: 40555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.697949] LNetError: 40555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.721198] LNet: Removed LNI 192.168.202.10@tcp [ 785.747776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.809940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.10@tcp [ 785.839676] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 786.466275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.10@tcp [ 786.491923] LNet: Removed LNI 192.168.202.10@tcp [ 786.547877] LNet: 40690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.166700] Key type .llcrypt unregistered [ 787.168658] Key type ._llcrypt unregistered [ 798.643397] Key type ._llcrypt registered [ 798.648972] Key type .llcrypt registered [ 799.962762] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 10:52:33 (1787842353) [ 802.230772] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg210-server [ 804.117274] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 10:52:37 (1787842357) [ 804.560318] Key type .llcrypt unregistered [ 804.562000] Key type ._llcrypt unregistered [ 811.164148] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 811.474914] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 811.481667] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 811.500531] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 814.002893] Key type ._llcrypt registered [ 814.004694] Key type .llcrypt registered [ 814.167435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 814.178444] alg: No test for adler32 (adler32-zlib) [ 815.065463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.113884] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 815.788510] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 817.528408] LNet: 42535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.533239] LNetError: 42535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.536360] LNet: Removed LNI 192.168.202.10@tcp1 [ 817.544962] LNet: Removed LNI 10.1.2.3@tcp2 [ 818.158422] Key type .llcrypt unregistered [ 818.162039] Key type ._llcrypt unregistered [ 829.992916] Key type ._llcrypt registered [ 829.995646] Key type .llcrypt registered [ 831.547267] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 10:53:04 (1787842384) [ 831.950357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 831.958906] alg: No test for adler32 (adler32-zlib) [ 832.956051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.018644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 833.038199] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 833.752441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 833.779450] LNet: 43508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.794115] LNetError: 43508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.810700] LNet: Removed LNI 192.168.202.10@tcp [ 833.846644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.908779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 833.939647] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 840.250207] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 10:53:13 (1787842393) [ 840.366631] LNet: 44091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.371923] LNetError: 44091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.388905] LNet: Removed LNI 192.168.202.10@tcp [ 840.990555] Key type .llcrypt unregistered [ 840.992992] Key type ._llcrypt unregistered [ 846.956867] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 847.176338] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 847.190789] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 847.216537] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 849.813202] Key type ._llcrypt registered [ 849.820201] Key type .llcrypt registered [ 850.029777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 850.054532] alg: No test for adler32 (adler32-zlib) [ 850.986163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.059360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 851.087919] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 851.762891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 851.799210] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 853.878386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 853.899239] LNet: 44839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.903748] LNetError: 44839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 853.914625] LNet: Removed LNI 10.1.2.3@tcp [ 853.934083] LNet: Removed LNI 192.168.202.10@tcp [ 853.955622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.012705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 854.044425] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 854.048162] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 854.484835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 854.510253] LNet: 44866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.516902] LNetError: 44866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 854.521383] LNetError: 44866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 854.530585] LNet: Removed LNI 10.1.2.3@tcp [ 854.543853] LNet: Removed LNI 192.168.202.10@tcp [ 854.569923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.656665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 854.678324] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 854.691569] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 855.082392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 855.098799] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 856.064823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 856.087893] LNet: 45028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.093328] LNetError: 45028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 856.099424] LNetError: 45028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 856.104709] LNet: Removed LNI 10.1.2.3@tcp [ 856.107259] LNet: Removed LNI 192.168.202.10@tcp [ 856.120505] LNet: Removed LNI 192.168.202.10@tcp2 [ 856.134953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.197873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 856.225033] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 856.233386] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 856.238822] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 864.170738] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 10:53:37 (1787842417) [ 864.275201] LNet: 45621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.281218] LNetError: 45621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 864.288563] LNetError: 45621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 864.294472] LNet: Removed LNI 10.1.2.3@tcp [ 864.296516] LNet: Removed LNI 192.168.202.10@tcp [ 864.310252] LNet: Removed LNI 192.168.202.10@tcp2 [ 864.835135] Key type .llcrypt unregistered [ 864.838768] Key type ._llcrypt unregistered [ 870.614034] Key type ._llcrypt registered [ 870.615413] Key type .llcrypt registered [ 870.847630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 870.869696] alg: No test for adler32 (adler32-zlib) [ 871.901622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.980640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 872.015088] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 872.705667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.751235] LNet: 46024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.760337] LNetError: 46024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.779494] LNet: Removed LNI 192.168.202.10@tcp [ 872.793432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.862711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 872.895309] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 873.463749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.498341] LNet: 46054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.510585] LNetError: 46054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.522753] LNet: Removed LNI 192.168.202.10@tcp [ 873.542700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.645457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 873.701059] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 874.191533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.228193] LNet: Removed LNI 192.168.202.10@tcp [ 874.255522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.306336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 874.328199] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 874.715134] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 874.891911] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 874.902553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 874.909190] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 877.272267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.323953] LNet: 46165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.330588] LNet: 46165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 877.350268] LNetError: 46165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.359854] LNetError: 46165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 877.383845] LNet: Removed LNI 192.168.202.10@tcp [ 877.423324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.477445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 877.493709] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 884.157245] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 10:53:57 (1787842437) [ 884.239827] LNet: 46754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.244642] LNetError: 46754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.258031] LNet: Removed LNI 192.168.202.10@tcp [ 884.782797] Key type .llcrypt unregistered [ 884.784792] Key type ._llcrypt unregistered [ 891.047251] Key type ._llcrypt registered [ 891.049678] Key type .llcrypt registered [ 891.284704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 891.302488] alg: No test for adler32 (adler32-zlib) [ 892.357543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.457218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 892.498555] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 893.470502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.484693] LNet: 47164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.488310] LNetError: 47164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.503368] LNet: Removed LNI 192.168.202.10@tcp [ 893.522914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.616293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 893.661726] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 900.373891] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 10:54:13 (1787842453) [ 900.547471] LNet: 47753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.557516] LNetError: 47753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.589825] LNet: Removed LNI 192.168.202.10@tcp [ 901.337721] Key type .llcrypt unregistered [ 901.351769] Key type ._llcrypt unregistered [ 907.843652] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 907.994993] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 908.010268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 908.018776] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 910.838741] Key type ._llcrypt registered [ 910.841606] Key type .llcrypt registered [ 911.045803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 911.062219] alg: No test for adler32 (adler32-zlib) [ 911.995586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.051531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 912.083284] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 913.095650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.134485] LNet: 48212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.141473] LNetError: 48212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.156902] LNet: Removed LNI 192.168.202.10@tcp [ 913.173029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.261434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 913.297874] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 920.237313] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 10:54:33 (1787842473) [ 920.319497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.353782] LNet: 48804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.362142] LNetError: 48804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.382198] LNet: Removed LNI 192.168.202.10@tcp [ 920.437057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.543287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 926.489455] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 10:54:39 (1787842479) [ 926.650185] LNet: 49387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.221589] Key type .llcrypt unregistered [ 927.224159] Key type ._llcrypt unregistered [ 934.472081] Key type ._llcrypt registered [ 934.476648] Key type .llcrypt registered [ 934.670911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 934.682932] alg: No test for adler32 (adler32-zlib) [ 935.689992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.754123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 935.779866] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 936.497037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.511513] LNet: 49793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.516483] LNetError: 49793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.524930] LNet: Removed LNI 192.168.202.10@tcp [ 936.540739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.586957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 936.604861] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 942.753695] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 10:54:56 (1787842496) [ 943.055142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.085606] LNet: 50386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.092675] LNetError: 50386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.114047] LNet: Removed LNI 192.168.202.10@tcp [ 943.153135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.276590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 949.525959] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 10:55:02 (1787842502) [ 950.886235] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 952.946905] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 10:55:05 (1787842505) [ 954.418401] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 956.242370] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 10:55:09 (1787842509) [ 958.110951] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 959.902183] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 10:55:13 (1787842513) [ 961.233577] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 962.807140] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 10:55:16 (1787842516) [ 964.543195] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 967.018534] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 10:55:19 (1787842519) [ 967.124223] LNet: 52429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.138905] Key type .llcrypt unregistered [ 968.141897] Key type ._llcrypt unregistered [ 975.482172] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 975.730861] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 975.740187] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 975.747164] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 978.453466] Key type ._llcrypt registered [ 978.455543] Key type .llcrypt registered [ 978.669542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 978.693255] alg: No test for adler32 (adler32-zlib) [ 979.752702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.887158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 979.919343] LNet: Added LNI 192.168.202.10@tcp [8/512/0/180] [ 979.938231] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 979.940071] LNet: 52758:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 980.058964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.083152] LNet: 52777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.094560] LNetError: 52777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 980.108654] LNet: Removed LNI 10.1.2.3@tcp [ 980.111397] LNetError: 52777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.126607] LNet: Removed LNI 192.168.202.10@tcp [ 980.151636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.245366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 980.278698] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 980.378095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.407530] LNet: 52801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.416915] LNetError: 52801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.434742] LNet: Removed LNI 192.168.202.10@tcp [ 980.458512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.550259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 980.572973] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 980.580264] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 980.584127] LNet: 52809:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 980.691881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.734619] LNet: 52827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.738074] LNetError: 52827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 980.748840] LNet: Removed LNI 10.1.2.3@tcp [ 980.752227] LNetError: 52827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.773857] LNet: Removed LNI 192.168.202.10@tcp [ 980.789280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.843225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 980.857698] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 980.862403] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 980.864353] LNet: 52835:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 981.073232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.107559] LNet: 52861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.116765] LNetError: 52861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 981.131658] LNet: Removed LNI 10.1.2.3@tcp [ 981.136517] LNetError: 52861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.147789] LNet: Removed LNI 192.168.202.10@tcp [ 981.174456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.268143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 981.309208] LNet: Added LNI 192.168.202.10@tcp [8/256/0/182] [ 981.316077] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 981.430393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.462719] LNet: 52889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.468743] LNetError: 52889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 981.474677] LNet: Removed LNI 10.1.2.3@tcp [ 981.477534] LNetError: 52889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.498427] LNet: Removed LNI 192.168.202.10@tcp [ 981.515314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.568545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 981.588624] LNet: Added LNI 192.168.202.10@tcp [8/256/0/182] [ 981.598855] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 981.602672] LNet: 52897:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 981.670666] LNet: Removed LNI 10.1.2.3@tcp [ 981.682505] LNet: Removed LNI 192.168.202.10@tcp [ 982.364942] Key type .llcrypt unregistered [ 982.369400] Key type ._llcrypt unregistered [ 994.059417] Key type ._llcrypt registered [ 994.065686] Key type .llcrypt registered [ 995.537106] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 10:55:48 (1787842548) [ 996.779198] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 998.104597] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:55:51 (1787842551) [ 998.688958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.702814] alg: No test for adler32 (adler32-zlib) [ 999.676611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.735585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 999.753614] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1000.443785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.10 [ 1000.459898] LNet: 54167:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1000.467082] LNet: Removed LNI 192.168.202.10@tcp [ 1006.651721] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:56:00 (1787842560) [ 1006.919187] LNet: 54737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.926420] LNetError: 54737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.939111] LNet: Removed LNI 192.168.202.10@tcp [ 1007.573540] Key type .llcrypt unregistered [ 1007.575417] Key type ._llcrypt unregistered [ 1014.860942] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1015.097690] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1016.692855] Key type ._llcrypt registered [ 1016.695826] Key type .llcrypt registered [ 1016.856995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1016.870905] alg: No test for adler32 (adler32-zlib) [ 1018.556674] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1025.139217] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:56:18 (1787842578) [ 1025.397809] LNet: 55647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.403496] LNetError: 55647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1025.421988] LNet: Removed LNI 10.1.2.3@tcp [ 1026.119519] Key type .llcrypt unregistered [ 1026.136956] Key type ._llcrypt unregistered [ 1033.302213] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1033.419491] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1033.421446] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1033.425775] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1036.046033] Key type ._llcrypt registered [ 1036.048280] Key type .llcrypt registered [ 1036.184026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1036.200672] alg: No test for adler32 (adler32-zlib) [ 1037.082209] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1044.550506] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:56:37 (1787842597) [ 1046.002753] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1047.964961] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:56:41 (1787842601) [ 1048.150172] LNet: 56836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.160080] LNetError: 56836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1048.169552] LNet: Removed LNI 10.1.2.3@tcp [ 1048.567510] Key type .llcrypt unregistered [ 1048.570169] Key type ._llcrypt unregistered [ 1056.026508] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1056.250240] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1056.253683] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1056.263654] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1059.266675] Key type ._llcrypt registered [ 1059.270640] Key type .llcrypt registered [ 1059.535317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1059.554768] alg: No test for adler32 (adler32-zlib) [ 1060.582732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.666489] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1068.412322] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:57:01 (1787842621) [ 1069.020854] LNet: 58069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.027436] LNetError: 58069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1069.050708] LNet: Removed LNI 10.1.2.3@tcp [ 1069.811919] Key type .llcrypt unregistered [ 1069.813972] Key type ._llcrypt unregistered [ 1076.465519] Key type ._llcrypt registered [ 1076.468593] Key type .llcrypt registered [ 1076.675274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1076.692867] alg: No test for adler32 (adler32-zlib) [ 1077.651927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1077.688184] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1085.308420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1085.915473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_interrupt -r 1 [ 1085.945994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_interrupt -r 1 [ 1085.971440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1085.985145] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1086.119879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1086.142320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1086.167988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1086.196347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1086.221258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1086.242296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1086.265126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1086.288508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1086.308359] LNet: 58511:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1086.322477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1086.350510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1086.378841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1086.404965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1086.441544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1086.469395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1086.495332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1086.958945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_dropped -r 1 [ 1087.011150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_dropped -r 1 [ 1087.046623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1087.069420] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1087.264683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1087.296331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1087.327991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1087.350429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1087.373609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1087.396142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1087.417789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1087.441556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1087.460562] LNet: 58620:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1087.474331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1087.497697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1087.527473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1087.543927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1087.566570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1087.596286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1087.627689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1088.024103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_aborted -r 1 [ 1088.041976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_aborted -r 1 [ 1088.077023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1088.092964] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1088.252530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1088.274261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1088.303761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1088.332949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1088.360707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1088.399336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1088.427367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1088.447750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1088.463470] LNet: 58729:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1088.477650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1088.510827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1088.543298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1088.564801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1088.583639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1088.601705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1088.625192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1089.030942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_no_route -r 1 [ 1089.058654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_no_route -r 1 [ 1089.093309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1089.312752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1089.340303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1089.366789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1089.387182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1089.407635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1089.429375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1089.450976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1089.471690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1089.496369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1089.517615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1089.532791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1089.556302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1089.587561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1089.611813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1089.637557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1090.101640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_timeout -r 1 [ 1090.134600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_timeout -r 1 [ 1090.167397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1090.190330] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1090.200682] LNet: Skipped 1 previous similar message [ 1090.382602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1090.415479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1090.437578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1090.466405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1090.502187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1090.537739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1090.586583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1090.615909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1090.637938] LNet: 58947:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1090.646220] LNet: 58947:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1090.673780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1090.717599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1090.749653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1090.783705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1090.824514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1090.862134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1090.903638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1091.347971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_error -r 1 [ 1091.387196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_error -r 1 [ 1091.426551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1091.628694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1091.658284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1091.700665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1091.742516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1091.792389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1091.827557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1091.867756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1091.918777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1091.956227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1091.993769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1092.027836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1092.065023] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1092.076270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1092.098630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1092.125458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1092.155374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1092.357189] LNet: 59102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1092.362369] LNetError: 59102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1092.372823] LNet: Removed LNI 192.168.202.10@tcp [ 1092.667723] Key type .llcrypt unregistered [ 1092.673960] Key type ._llcrypt unregistered [ 1108.803833] Key type ._llcrypt registered [ 1108.813326] Key type .llcrypt registered [ 1110.020138] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:57:43 (1787842663) [ 1110.921376] Key type .llcrypt unregistered [ 1110.922819] Key type ._llcrypt unregistered [ 1116.773865] Key type ._llcrypt registered [ 1116.775795] Key type .llcrypt registered [ 1116.934983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1116.946971] alg: No test for adler32 (adler32-zlib) [ 1117.901213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1117.929574] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1125.497873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1135.575360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1135.613460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1135.638710] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1136.663851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_interrupt -r 1 [ 1136.689129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_interrupt -r 1 [ 1136.720697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_interrupt -r 1 [ 1136.773493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_interrupt -r 1 [ 1136.841049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_interrupt -r 1 [ 1136.894525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_interrupt -r 1 [ 1136.934440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_interrupt -r 1 [ 1136.982595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_interrupt -r 1 [ 1137.010058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1137.029157] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1137.033616] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1137.182640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1137.199769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1137.218738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1137.237168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1137.256848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1137.279365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1137.302673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1137.326884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1137.343959] LNet: 60572:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1137.358982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1137.399648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1137.451164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1137.528484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1137.590394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.634192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1137.693139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1138.316227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_dropped -r 1 [ 1138.362364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_dropped -r 1 [ 1138.425915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_dropped -r 1 [ 1138.473102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_dropped -r 1 [ 1138.549935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_dropped -r 1 [ 1138.616307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_dropped -r 1 [ 1138.687995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_dropped -r 1 [ 1138.732483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_dropped -r 1 [ 1138.765812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1138.794482] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1138.804566] LNet: Skipped 2 previous similar messages [ 1139.002467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1139.038162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1139.085839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1139.127084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1139.157741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1139.188508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1139.253801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1139.313833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1139.363870] LNet: 60693:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1139.386138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1139.444553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1139.469439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1139.496754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1139.524858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1139.554774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1139.576625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1139.998983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_aborted -r 1 [ 1140.029675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_aborted -r 1 [ 1140.062623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_aborted -r 1 [ 1140.088925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_aborted -r 1 [ 1140.118826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_aborted -r 1 [ 1140.146985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_aborted -r 1 [ 1140.170906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_aborted -r 1 [ 1140.210519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_aborted -r 1 [ 1140.241326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1140.257869] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1140.261867] LNet: Skipped 2 previous similar messages [ 1140.455106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.474758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1140.496553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1140.523638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1140.555600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.588579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1140.615994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1140.654716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1140.683247] LNet: 60815:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1140.720735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1140.750568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1140.782200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1140.813884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.844770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1140.871992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1140.906447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1141.523448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_no_route -r 1 [ 1141.570221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_no_route -r 1 [ 1141.636816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_no_route -r 1 [ 1141.682756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_no_route -r 1 [ 1141.760860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_no_route -r 1 [ 1141.806439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_no_route -r 1 [ 1141.842859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_no_route -r 1 [ 1141.870553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_no_route -r 1 [ 1141.897621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1142.106454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1142.150467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1142.175470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1142.197219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1142.223789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1142.243242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1142.261767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1142.287454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1142.316494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1142.347078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1142.385757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1142.428677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1142.485075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1142.521217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1142.552065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1143.080349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_timeout -r 1 [ 1143.132605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_timeout -r 1 [ 1143.170142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_timeout -r 1 [ 1143.197382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_timeout -r 1 [ 1143.239363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_timeout -r 1 [ 1143.291732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_timeout -r 1 [ 1143.343782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_timeout -r 1 [ 1143.392741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_timeout -r 1 [ 1143.452993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1143.473223] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1143.480022] LNet: Skipped 5 previous similar messages [ 1143.737710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1143.761525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1143.791260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1143.814571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1143.846928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1143.878300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1143.913833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1143.959590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1143.976957] LNet: 61057:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1143.982445] LNet: 61057:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1143.998768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1144.030235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1144.054696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1144.080140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1144.105545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1144.138907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1144.179974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1144.687464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_error -r 1 [ 1144.715786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_error -r 1 [ 1144.745863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_error -r 1 [ 1144.772587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_error -r 1 [ 1144.801395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_error -r 1 [ 1144.829936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_error -r 1 [ 1144.858389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_error -r 1 [ 1144.907443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_error -r 1 [ 1144.954775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1145.163830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1145.198224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1145.228273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1145.255817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1145.283928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1145.318049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1145.347533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1145.378464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1145.406353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1145.471617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1145.521810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1145.552774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1145.580596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1145.603930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1145.628869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1150.650186] LNet: 61258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1150.656691] LNetError: 61258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1150.674849] LNet: Removed LNI 192.168.202.10@tcp [ 1151.728368] LNet: Removed LNI 192.168.202.10@tcp1 [ 1152.119802] Key type .llcrypt unregistered [ 1152.124206] Key type ._llcrypt unregistered [ 1167.164591] Key type ._llcrypt registered [ 1167.167839] Key type .llcrypt registered [ 1168.475284] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:58:41 (1787842721) [ 1169.382486] Key type .llcrypt unregistered [ 1169.385511] Key type ._llcrypt unregistered [ 1176.423048] Key type ._llcrypt registered [ 1176.427160] Key type .llcrypt registered [ 1176.684736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1176.699568] alg: No test for adler32 (adler32-zlib) [ 1177.692830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1177.714760] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1185.441775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1186.260536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_dropped -r 1 [ 1186.289802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_dropped -r 1 [ 1186.318500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1186.336408] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1186.537424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.568461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1186.595213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1186.630872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1186.661662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1186.682340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1186.697890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1186.712868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.727656] LNet: 62510:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1186.745375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.772313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.795704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1186.819395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1186.846366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.874088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1186.920200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1187.401317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_error -r 1 [ 1187.428761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_error -r 1 [ 1187.458730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1187.474521] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1187.619927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1187.645840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1187.671169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1187.700985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1187.723573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1187.746345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1187.770542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1187.797392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1187.819866] LNet: 62619:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1187.835202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1187.860850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1187.890503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1187.908924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1187.923424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1187.940699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1187.955985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1188.402616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_timeout -r 1 [ 1188.440087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_timeout -r 1 [ 1188.468969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1188.485758] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1188.685840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1188.708706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1188.734404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1188.768597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1188.813894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1188.840722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1188.865809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1188.895655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1188.910083] LNet: 62729:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1188.923680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1188.957050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1188.993581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1189.046266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1189.077988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.109977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.141699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1189.396387] LNet: 62774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1189.401756] LNetError: 62774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1189.416642] LNet: Removed LNI 192.168.202.10@tcp [ 1189.947682] Key type .llcrypt unregistered [ 1189.950194] Key type ._llcrypt unregistered [ 1207.609324] Key type ._llcrypt registered [ 1207.611972] Key type .llcrypt registered [ 1208.874910] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:59:22 (1787842762) [ 1209.738450] Key type .llcrypt unregistered [ 1209.739562] Key type ._llcrypt unregistered [ 1215.599941] Key type ._llcrypt registered [ 1215.602369] Key type .llcrypt registered [ 1215.823443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1215.851792] alg: No test for adler32 (adler32-zlib) [ 1216.733364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1216.755543] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1223.395935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1232.192211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1232.251801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1232.295904] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1233.596196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_dropped -r 1 [ 1233.649650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e remote_dropped -r 1 [ 1233.690620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_dropped -r 1 [ 1233.733916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e remote_dropped -r 1 [ 1233.779294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e remote_dropped -r 1 [ 1233.811880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e remote_dropped -r 1 [ 1233.848430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e remote_dropped -r 1 [ 1233.887346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e remote_dropped -r 1 [ 1233.922870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1233.938669] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1233.943939] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 1234.101952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1234.119716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1234.159137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1234.183664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1234.215662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1234.251976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1234.287944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1234.319029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1234.336357] LNet: 64244:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1234.366831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1234.396776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.429485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1234.456678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1234.489351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1234.534172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1234.578832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1234.610698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1235.197529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_error -r 1 [ 1235.231276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e remote_error -r 1 [ 1235.257918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_error -r 1 [ 1235.288809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e remote_error -r 1 [ 1235.317492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e remote_error -r 1 [ 1235.350136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e remote_error -r 1 [ 1235.383899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e remote_error -r 1 [ 1235.418338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e remote_error -r 1 [ 1235.463853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1235.484425] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1235.489638] LNet: Skipped 2 previous similar messages [ 1235.696907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1235.727866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1235.778964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1235.822557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1235.857526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1235.881757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1235.933714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.005828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.032692] LNet: 64367:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.047545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.080853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.116606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.147829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.168559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.195250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1236.219417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1236.246569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.698248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_timeout -r 1 [ 1236.743726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e remote_timeout -r 1 [ 1236.777916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_timeout -r 1 [ 1236.821722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e remote_timeout -r 1 [ 1236.877717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e remote_timeout -r 1 [ 1236.914311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e remote_timeout -r 1 [ 1236.942917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e remote_timeout -r 1 [ 1236.998215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e remote_timeout -r 1 [ 1237.038273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1237.069821] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1237.075881] LNet: Skipped 1 previous similar message [ 1237.312822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1237.354912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1237.383673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1237.422777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1237.449985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1237.481369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1237.504246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1237.537401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1237.558905] LNet: 64490:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1237.582211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1237.610798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1237.645596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1237.690571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1237.716794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1237.746491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1237.795748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1237.839946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1242.067356] LNet: 64572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1242.079476] LNetError: 64572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1242.085328] LNet: Removed LNI 192.168.202.10@tcp [ 1242.111676] LNet: Removed LNI 192.168.202.10@tcp1 [ 1242.582690] Key type .llcrypt unregistered [ 1242.593428] Key type ._llcrypt unregistered [ 1261.302754] Key type ._llcrypt registered [ 1261.304593] Key type .llcrypt registered [ 1262.448594] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 11:00:15 (1787842815) [ 1264.002529] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1265.834576] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 11:00:19 (1787842819) [ 1266.680402] Key type .llcrypt unregistered [ 1266.684235] Key type ._llcrypt unregistered [ 1273.128842] Key type ._llcrypt registered [ 1273.131535] Key type .llcrypt registered [ 1273.290997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1273.302710] alg: No test for adler32 (adler32-zlib) [ 1274.311585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1274.334373] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1282.696133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1283.700926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e network_timeout -r 1 [ 1283.740666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e network_timeout -r 1 [ 1283.785510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1283.810412] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1284.124399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1284.162656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1284.208897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1284.245990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1284.285612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1284.321424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1284.354209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1284.389804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1284.408370] LNet: 66119:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1284.426905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1284.461817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1284.496237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1284.520900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1284.550411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1284.576398] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1284.584697] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1284.594598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1284.986427] LNet: 66162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1284.997800] LNetError: 66162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1285.025146] LNet: Removed LNI 192.168.202.10@tcp [ 1285.570554] Key type .llcrypt unregistered [ 1285.572583] Key type ._llcrypt unregistered [ 1305.189247] Key type ._llcrypt registered [ 1305.194287] Key type .llcrypt registered [ 1305.575368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1305.619948] alg: No test for adler32 (adler32-zlib) [ 1306.649944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1306.700495] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1315.326660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1324.873978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.924980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1324.953796] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1326.523684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e network_timeout -r 1 [ 1326.559977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e network_timeout -r 1 [ 1326.600252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e network_timeout -r 1 [ 1326.645756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e network_timeout -r 1 [ 1326.702750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e network_timeout -r 1 [ 1326.753697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e network_timeout -r 1 [ 1326.829889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e network_timeout -r 1 [ 1326.895950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e network_timeout -r 1 [ 1326.966144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1327.001171] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1327.204646] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1327.219181] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 1327.297569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1327.323406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1327.360845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1327.402512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1327.446936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1327.475267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1327.512227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1327.542466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1327.559517] LNet: 67058:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1327.578073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1327.616732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1327.643551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1327.679288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1327.720698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1327.758831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1332.571384] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1332.943307] LNet: 67136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1332.955431] LNetError: 67136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.969324] LNet: Removed LNI 192.168.202.10@tcp [ 1332.985347] LNet: Removed LNI 192.168.202.10@tcp1 [ 1333.319716] Key type .llcrypt unregistered [ 1333.323245] Key type ._llcrypt unregistered [ 1350.917833] Key type ._llcrypt registered [ 1350.922196] Key type .llcrypt registered [ 1352.488862] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 11:01:45 (1787842905) [ 1353.194409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1353.213881] alg: No test for adler32 (adler32-zlib) [ 1354.137665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1354.184876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1354.201313] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1354.858413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1354.894425] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1355.767693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1355.841644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1355.906818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1355.930952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1355.975803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1356.021751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1356.055931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.10@tcp [ 1356.077026] LNetError: 68236:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.10@tcp to recovery [ 1356.192402] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1356.195333] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1358.240413] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1358.560872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1362.336339] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1363.294263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1363.337890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1363.383808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1363.396879] LNet: 68343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1363.407391] LNetError: 68343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1363.413929] LNet: Removed LNI 192.168.202.10@tcp [ 1363.422615] LNet: Removed LNI 192.168.202.10@tcp1 [ 1363.451784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1363.539916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1363.571981] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1364.359913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1364.391842] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1365.202763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1365.245305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1365.315440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1365.368347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1365.406932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1365.444466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1365.476401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1365.511150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.10@tcp [ 1365.530642] LNetError: 68616:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.10@tcp to recovery [ 1365.549118] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1368.092364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1371.681173] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1371.695219] LNet: Skipped 1 previous similar message [ 1376.137224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1376.170395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1376.226456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1383.216631] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 11:02:16 (1787842936) [ 1383.338469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1383.369276] LNet: 69315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.376445] LNetError: 69315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.387688] LNetError: 69315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1383.410333] LNet: Removed LNI 192.168.202.10@tcp [ 1383.435748] LNet: Removed LNI 192.168.202.10@tcp1 [ 1383.495941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1383.546953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1383.564760] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1384.146776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1384.162800] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1384.916449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1384.978194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1385.007472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1385.041001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1385.077540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1385.127635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1385.154768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.10@tcp --health 0 [ 1385.316576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1385.568222] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.10@tcp [ 1386.592448] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1386.598121] LNet: Skipped 1 previous similar message [ 1402.061601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1402.221030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1402.247296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1402.288606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.10@tcp --health 500 [ 1402.406641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1409.192697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1409.253287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1409.284602] LNet: 69823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1409.300162] LNetError: 69823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1409.312163] LNetError: 69823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1410.336172] LNet: Removed LNI 192.168.202.10@tcp [ 1410.348790] LNet: Removed LNI 192.168.202.10@tcp1 [ 1410.408822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1410.473565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1410.497040] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1411.192407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1411.214051] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1412.228708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1412.283740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1412.362078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1412.391800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1412.427847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1412.472713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1412.510662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1412.573059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.10@tcp --health 0 [ 1413.536557] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1413.551103] LNet: Skipped 3 previous similar messages [ 1413.953276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1424.255496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1424.303946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1424.340806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1431.173639] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 11:03:04 (1787842984) [ 1431.594861] LNet: 70871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.600702] LNetError: 70871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.605990] LNetError: 70871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1431.612160] LNet: Removed LNI 192.168.202.10@tcp [ 1431.622713] LNet: Removed LNI 192.168.202.10@tcp1 [ 1432.211900] Key type .llcrypt unregistered [ 1432.214699] Key type ._llcrypt unregistered [ 1438.295360] Key type ._llcrypt registered [ 1438.303887] Key type .llcrypt registered [ 1438.562310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1438.582617] alg: No test for adler32 (adler32-zlib) [ 1439.461932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1439.486234] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1449.085759] Lustre: DEBUG MARKER: Initial discovery [ 1449.409938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.110@tcp [ 1451.487671] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1451.754435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1451.789763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.110@tcp [ 1451.823043] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1451.851565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1451.904914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.110@tcp -r 1 -m GET -l 3 [ 1451.982278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.110@tcp -r 1 -m GET -e local_error [ 1452.042309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.110@tcp -r 1 -m PUT -l 6 [ 1452.832326] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1453.516487] Lustre: DEBUG MARKER: Force oleg210-client.virtnet to discover 192.168.202.110@tcp (in background) [ 1453.677916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.110@tcp [ 1455.000297] Lustre: DEBUG MARKER: Force oleg210-server to discover 192.168.202.10@tcp [ 1456.803594] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1457.404213] Lustre: DEBUG MARKER: Wait for 71634 [ 1459.564660] Lustre: DEBUG MARKER: Finished wait on 71634 [ 1459.614875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1459.678276] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1459.680023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1460.039404] LNet: 72091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1460.046973] LNetError: 72091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1460.092187] LNet: Removed LNI 192.168.202.10@tcp [ 1460.627791] Key type .llcrypt unregistered [ 1460.630388] Key type ._llcrypt unregistered [ 1478.511625] Key type ._llcrypt registered [ 1478.513471] Key type .llcrypt registered [ 1479.980404] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 11:03:52 (1787843032) [ 1480.668537] Key type .llcrypt unregistered [ 1480.673249] Key type ._llcrypt unregistered [ 1487.241224] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1487.403455] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1487.416089] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1487.427855] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1490.301240] Key type ._llcrypt registered [ 1490.303354] Key type .llcrypt registered [ 1490.503556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1490.542603] alg: No test for adler32 (adler32-zlib) [ 1491.474439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1491.549897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1491.576476] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1492.295911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1492.313361] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1499.949290] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 11:04:13 (1787843053) [ 1500.165172] LNet: 74176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.179496] LNetError: 74176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1500.186083] LNet: Removed LNI 10.1.2.3@tcp [ 1500.196362] LNet: Removed LNI 192.168.202.10@tcp [ 1500.748706] Key type .llcrypt unregistered [ 1500.755996] Key type ._llcrypt unregistered [ 1507.750107] Key type ._llcrypt registered [ 1507.754514] Key type .llcrypt registered [ 1507.939890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1507.954947] alg: No test for adler32 (adler32-zlib) [ 1508.879302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1508.936938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1508.956837] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1509.719728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1509.745694] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1511.109089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.10@tcp [ 1512.162647] LNetError: Unexpected error -22 connecting to 192.168.202.10@tcp at host 192.168.202.10:988 [ 1518.663750] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 11:04:31 (1787843071) [ 1519.094552] LNet: 75311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.100154] LNetError: 75311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1519.106620] LNet: Removed LNI 10.1.2.3@tcp [ 1519.122454] LNet: Removed LNI 192.168.202.10@tcp [ 1519.655596] Key type .llcrypt unregistered [ 1519.657120] Key type ._llcrypt unregistered [ 1528.849292] Key type ._llcrypt registered [ 1528.850817] Key type .llcrypt registered [ 1529.352300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1529.393349] alg: No test for adler32 (adler32-zlib) [ 1530.415838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.545318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1530.596557] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1531.821902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1531.847464] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 1533.081924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.10@tcp1 --nid 192.168.202.10@tcp2 [ 1540.155837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp1 192.168.202.10@tcp2 [ 1540.177985] LNetError: 76152:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.10@tcp1 to 192.168.202.10@tcp2 - Route table may be misconfigured [ 1540.223449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp2 192.168.202.10@tcp1 [ 1548.105869] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 11:05:01 (1787843101) [ 1548.285360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.315372] LNet: 76719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.327096] LNetError: 76719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1548.344325] LNet: Removed LNI 192.168.202.10@tcp1 [ 1549.421243] LNet: Removed LNI 192.168.202.10@tcp2 [ 1549.497145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.649361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1549.727421] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1550.638633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1550.662637] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1551.553579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1551.605644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.10@tcp [ 1551.633994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp1 192.168.202.10@tcp1 [ 1551.679544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.10@tcp -d 192.168.202.10@tcp -e network_timeout [ 1551.714664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -e network_timeout [ 1551.770307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -e network_timeout [ 1551.823180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -e network_timeout [ 1551.867985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1551.891288] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1551.906877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1552.005607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1552.114942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1552.672298] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1558.300991] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 11:05:11 (1787843111) [ 1558.588400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1558.627838] LNet: 77586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1558.638191] LNetError: 77586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1558.652520] LNetError: 77586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1558.666525] LNet: Removed LNI 192.168.202.10@tcp [ 1559.722150] LNet: Removed LNI 192.168.202.10@tcp1 [ 1559.757606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.886303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1560.195842] LNet: 77628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.626961] Key type .llcrypt unregistered [ 1560.636742] Key type ._llcrypt unregistered [ 1571.458387] Key type ._llcrypt registered [ 1571.460093] Key type .llcrypt registered [ 1572.608216] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 11:05:25 (1787843125) [ 1573.010444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1573.026761] alg: No test for adler32 (adler32-zlib) [ 1573.989615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.258896] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1576.940379] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 11:05:30 (1787843130) [ 1577.301659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.339695] LNet: 78725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.362420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.423454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1577.452943] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1578.206528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1578.221869] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1578.937074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1578.969923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp1 [ 1579.001629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp1 [ 1584.839761] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 11:05:38 (1787843138) [ 1586.521029] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg210-server [ 1587.970432] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 11:05:41 (1787843141) [ 1590.004502] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg210-server [ 1591.428489] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 11:05:44 (1787843144) [ 1593.277479] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg210-server [ 1594.947039] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 11:05:48 (1787843148) [ 1597.156994] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg210-server [ 1598.917906] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 11:05:52 (1787843152) [ 1600.719493] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg210-server [ 1602.696399] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 11:05:55 (1787843155) [ 1604.761733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg210-server [ 1606.630821] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 11:05:59 (1787843159) [ 1609.239520] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg210-server [ 1610.497580] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 11:06:04 (1787843164) [ 1612.208945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg210-server [ 1613.613710] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 11:06:07 (1787843167) [ 1615.420357] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg210-server [ 1617.619627] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 11:06:10 (1787843170) [ 1617.751895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.797302] LNet: 82597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.807338] LNetError: 82597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.818321] LNet: Removed LNI 192.168.202.10@tcp [ 1618.866097] LNet: Removed LNI 192.168.202.10@tcp1 [ 1618.902545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.001777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1619.056354] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1620.004828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1620.072417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1620.342615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.386382] LNet: 82751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1620.393532] LNetError: 82751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1620.399320] LNetError: 82751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1621.419219] LNet: Removed LNI 192.168.202.10@tcp [ 1621.473383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.591848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1621.637347] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1622.548266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1622.748505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.769844] LNet: 82897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.777611] LNetError: 82897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.849471] LNet: Removed LNI 192.168.202.10@tcp [ 1623.903515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.001841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1624.050942] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1624.701743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1624.770685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1626.174866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1626.208255] LNetError: 83053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.229291] LNet: Removed LNI 192.168.202.10@tcp [ 1626.243704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.296317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1626.316406] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1627.325559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1627.591931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.620515] LNet: 83198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1627.626938] LNet: 83198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1628.711484] LNet: Removed LNI 192.168.202.10@tcp [ 1628.816681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.880806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1628.899831] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1629.715445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1629.803669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1630.032419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.082576] LNet: Removed LNI 192.168.202.10@tcp [ 1631.126429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.225022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1631.262228] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1632.256921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1633.726354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1633.753343] LNetError: 83500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.757619] LNetError: 83500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1634.800098] LNet: Removed LNI 192.168.202.10@tcp [ 1634.832313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.937757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1634.990132] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1636.048728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1636.141158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1636.341424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.373716] LNet: 83648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.379827] LNet: 83648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1637.418513] LNet: Removed LNI 192.168.202.10@tcp [ 1637.464555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.592992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1637.654324] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1638.566597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1638.867667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.976213] LNet: Removed LNI 192.168.202.10@tcp [ 1640.031666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.112300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1640.138522] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1641.104735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1641.171872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1641.405825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.471291] LNet: Removed LNI 192.168.202.10@tcp [ 1642.528722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.610400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1642.636082] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1643.407275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1643.678710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1643.713328] LNetError: 84086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1643.718110] LNetError: 84086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1644.780853] LNet: Removed LNI 192.168.202.10@tcp [ 1644.846445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.970259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1645.012425] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1645.695671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1645.789528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1647.273651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.364046] LNet: Removed LNI 192.168.202.10@tcp [ 1648.389965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.478976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1648.530843] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1649.251564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1649.546951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1650.673747] LNet: Removed LNI 192.168.202.10@tcp [ 1650.731869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.808874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1650.838234] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1651.630333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1651.698218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1651.728813] LNet: Removed LNI 192.168.202.10@tcp [ 1651.744374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.800277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1651.823079] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1652.591982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1652.694684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1652.730225] LNet: 84658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.733423] LNet: 84658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1652.755350] LNet: Removed LNI 192.168.202.10@tcp [ 1652.777622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.842475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1652.929291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1652.967298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.019605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1653.104169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.172221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.273202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1653.403392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.448919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.496395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1653.516252] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1654.164582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.192278] LNet: Removed LNI 192.168.202.10@tcp [ 1654.204835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.273592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1661.076834] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 11:06:54 (1787843214) [ 1661.204769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1661.305776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.419468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1661.453436] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 1663.033257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1663.067786] LNetError: 85549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1663.081253] LNetError: 85549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1663.106066] LNet: Removed LNI 192.168.202.10@tcp231 [ 1663.128037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.200705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1663.227222] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 1663.233099] LNet: 85557:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1663.721444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1663.778419] LNet: Removed LNI 192.168.202.10@tcp231 [ 1663.816642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1663.853149] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 1665.134606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1665.169888] LNet: Removed LNI 192.168.202.10@tcp231 [ 1665.191103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.272958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1665.299456] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/0] [ 1665.308898] LNet: 85709:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1665.672121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1665.715657] LNet: Removed LNI 192.168.202.10@tcp231 [ 1665.743061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1665.789357] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/0] [ 1667.105373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.145180] LNet: Removed LNI 192.168.202.10@tcp231 [ 1667.173784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.245286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1667.294288] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/60] [ 1667.303468] LNet: 85861:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1667.677352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1667.716088] LNet: Removed LNI 192.168.202.10@tcp231 [ 1667.748184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1667.794191] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/60] [ 1668.980382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1669.012129] LNet: Removed LNI 192.168.202.10@tcp231 [ 1669.031890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.110123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1669.140101] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 1669.500681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1669.531209] LNet: Removed LNI 192.168.202.10@tcp231 [ 1669.546981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1669.576784] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 1678.589284] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 11:07:11 (1787843231) [ 1678.815912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.912147] LNet: Removed LNI 192.168.202.10@tcp231 [ 1678.929207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.976656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1678.999714] LNet: Added LNI 192.168.202.10@tcp232 [8/256/0/180] [ 1679.689900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1679.740942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1679.793744] LNet: Removed LNI 192.168.202.10@tcp232 [ 1679.814799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1679.855586] LNet: Added LNI 192.168.202.10@tcp232 [8/256/0/180] [ 1680.538972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1680.593887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1680.644935] LNet: Removed LNI 192.168.202.10@tcp232 [ 1680.675924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1680.712225] LNet: Added LNI 192.168.202.10@tcp232 [8/256/0/180] [ 1681.501811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1681.555145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1681.589799] LNet: Removed LNI 192.168.202.10@tcp232 [ 1681.606426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1681.625433] LNet: Added LNI 192.168.202.10@tcp232 [8/256/0/180] [ 1682.479973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1682.506405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1682.535268] LNet: Removed LNI 192.168.202.10@tcp232 [ 1682.558880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1688.712587] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 11:07:21 (1787843241) [ 1689.431511] LNet: 87851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1689.445400] LNet: 87851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1690.215001] Key type .llcrypt unregistered [ 1690.221431] Key type ._llcrypt unregistered [ 1696.485048] Key type ._llcrypt registered [ 1696.488328] Key type .llcrypt registered [ 1696.746802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1696.774155] alg: No test for adler32 (adler32-zlib) [ 1697.688878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1697.712397] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1704.795437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1713.847721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.910868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1713.933848] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1715.264776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_interrupt -r 2 [ 1715.317393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_interrupt -r 2 [ 1715.380659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_interrupt -r 2 [ 1715.447721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_interrupt -r 2 [ 1715.487896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_interrupt -r 2 [ 1715.542776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_interrupt -r 2 [ 1715.584606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_interrupt -r 2 [ 1715.630586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_interrupt -r 2 [ 1715.691230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1715.711345] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1715.728479] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1715.998783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1716.022684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1716.046239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1716.071720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1716.091970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1716.110847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1716.140608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1716.181373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1716.208577] LNet: 88518:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1716.232760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1716.266430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1716.293301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1716.324836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1716.347150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1716.371418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1716.868359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_dropped -r 2 [ 1716.907634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_dropped -r 2 [ 1716.948421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_dropped -r 2 [ 1717.012586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_dropped -r 2 [ 1717.066749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_dropped -r 2 [ 1717.118257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_dropped -r 2 [ 1717.166516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_dropped -r 2 [ 1717.213284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_dropped -r 2 [ 1717.270818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1717.294822] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1717.538372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1717.564685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1717.600297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1717.629417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1717.652798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1717.677654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1717.703994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1717.728590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1717.743262] LNet: 88639:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1717.756937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1717.789333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1717.835697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1717.879460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1717.913474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1717.961889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1718.436679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_aborted -r 2 [ 1718.461879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_aborted -r 2 [ 1718.500647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_aborted -r 2 [ 1718.534356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_aborted -r 2 [ 1718.579367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_aborted -r 2 [ 1718.614920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_aborted -r 2 [ 1718.653453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_aborted -r 2 [ 1718.695274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_aborted -r 2 [ 1718.742095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1718.771547] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1719.070336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1719.112770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1719.153358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1719.201352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1719.254432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1719.289267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1719.331799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1719.367350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1719.395252] LNet: 88759:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1719.409547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1719.431454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1719.464085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1719.489881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1719.512472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1719.539564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1720.005792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_no_route -r 2 [ 1720.040784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_no_route -r 2 [ 1720.077506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_no_route -r 2 [ 1720.114405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_no_route -r 2 [ 1720.146722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_no_route -r 2 [ 1720.179972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_no_route -r 2 [ 1720.211801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_no_route -r 2 [ 1720.241947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_no_route -r 2 [ 1720.270680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1720.515730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1720.547582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1720.569811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1720.597248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1720.623081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1720.649887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1720.676286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1720.701236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1720.731515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1720.760643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1720.789571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1720.823145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1720.853227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1720.885478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1721.472466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_timeout -r 2 [ 1721.526244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_timeout -r 2 [ 1721.565664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_timeout -r 2 [ 1721.596723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_timeout -r 2 [ 1721.621373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_timeout -r 2 [ 1721.655870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_timeout -r 2 [ 1721.698421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_timeout -r 2 [ 1721.741580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_timeout -r 2 [ 1721.787787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1721.812107] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1721.819748] LNet: Skipped 1 previous similar message [ 1722.155471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1722.215910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1722.250636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1722.301050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1722.346899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1722.427022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1722.477792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1722.527132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1722.565676] LNet: 88999:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1722.580390] LNet: 88999:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1722.600652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1722.650626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1722.682529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1722.711661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1722.767952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1722.821098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1723.521452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_dropped -r 2 [ 1723.564783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e remote_dropped -r 2 [ 1723.617684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_dropped -r 2 [ 1723.654944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e remote_dropped -r 2 [ 1723.697970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e remote_dropped -r 2 [ 1723.733701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e remote_dropped -r 2 [ 1723.769241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e remote_dropped -r 2 [ 1723.810451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e remote_dropped -r 2 [ 1723.854345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1723.884561] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 1724.156771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1724.204144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1724.225493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1724.262753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1724.305654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1724.355434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1724.417218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1724.479248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1724.535649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1724.559245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1724.584674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1724.625841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1724.658499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1724.702468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1730.913457] LNet: 89198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.918951] LNetError: 89198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1730.933311] LNet: Removed LNI 192.168.202.10@tcp [ 1730.947944] LNet: Removed LNI 192.168.202.10@tcp1 [ 1731.358544] Key type .llcrypt unregistered [ 1731.360133] Key type ._llcrypt unregistered [ 1749.714298] Key type ._llcrypt registered [ 1749.716459] Key type .llcrypt registered [ 1751.358678] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 11:08:24 (1787843304) [ 1751.873364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1751.889504] alg: No test for adler32 (adler32-zlib) [ 1752.920424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.010584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1753.033979] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1754.094478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1754.134830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.10@tcp [ 1754.154956] LNetError: 90179:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.10@tcp to recovery [ 1754.291739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.10@tcp [ 1754.594244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1761.963524] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 11:08:35 (1787843315) [ 1762.052607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.084414] LNet: 90766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1762.097464] LNetError: 90766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.124927] LNet: Removed LNI 192.168.202.10@tcp [ 1762.141502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.184838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1762.204460] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1762.865509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1762.890152] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1763.664680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.10@tcp [ 1763.680455] LNetError: 91035:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.10@tcp to recovery [ 1764.194314] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1764.200717] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1764.917811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.10@tcp [ 1775.151924] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 11:08:48 (1787843328) [ 1777.165734] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg210-server [ 1779.102926] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 11:08:52 (1787843332) [ 1780.921659] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg210-server [ 1783.209970] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 11:08:56 (1787843336) [ 1785.234661] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg210-server [ 1787.894121] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 11:09:00 (1787843340) [ 1787.994585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.014920] LNet: 92659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1788.027154] LNetError: 92659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1788.040514] LNet: Removed LNI 192.168.202.10@tcp [ 1788.060386] LNet: Removed LNI 192.168.202.10@tcp1 [ 1788.094811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1788.202309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1788.269458] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1789.554384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1789.660134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1789.728485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1789.844040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1789.931930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1790.006520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1796.616985] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 11:09:09 (1787843349) [ 1796.709103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1796.729933] LNet: 93390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1796.739867] LNetError: 93390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1796.745579] LNetError: 93390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1796.784154] LNet: Removed LNI 192.168.202.10@tcp [ 1796.813780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1796.867827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1796.889412] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1803.740935] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 11:09:16 (1787843356) [ 1803.839142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.879975] LNet: 94090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.892734] LNetError: 94090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1803.924192] LNet: Removed LNI 192.168.202.10@tcp [ 1803.971954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.047970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1804.079279] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1810.516889] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 11:09:23 (1787843363) [ 1811.861754] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1813.564789] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 11:09:26 (1787843366) [ 1813.992441] LNet: 95131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.998100] LNetError: 95131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.021594] LNet: Removed LNI 192.168.202.10@tcp [ 1815.054896] Key type .llcrypt unregistered [ 1815.058743] Key type ._llcrypt unregistered [ 1822.127948] Key type ._llcrypt registered [ 1822.129679] Key type .llcrypt registered [ 1822.592810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1822.645907] alg: No test for adler32 (adler32-zlib) [ 1823.606644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1823.672125] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1833.700749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1839.956340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.110@tcp [ 1840.369983] LNet: 95557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.375758] LNetError: 95557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.390967] LNet: Removed LNI 192.168.202.10@tcp [ 1840.999678] Key type .llcrypt unregistered [ 1841.004185] Key type ._llcrypt unregistered [ 1854.106827] Key type ._llcrypt registered [ 1854.108872] Key type .llcrypt registered [ 1855.241927] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1856.919436] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1858.751076] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 11:10:11 (1787843411) [ 1860.589032] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1862.535380] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 11:10:15 (1787843415) [ 1864.490487] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg210-server [ 1866.170489] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 11:10:19 (1787843419) [ 1867.879158] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg210-server [ 1869.642653] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 11:10:22 (1787843422) [ 1871.254189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1871.297241] alg: No test for adler32 (adler32-zlib) [ 1872.329878] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 1872.624220] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1874.288170] Key type lgssc registered [ 1875.813577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1896.246300] Key type lgssc unregistered [ 1896.512833] LNet: 98734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.516768] LNetError: 98734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.541714] LNet: Removed LNI 192.168.202.10@tcp [ 1897.240513] Key type .llcrypt unregistered [ 1897.242955] Key type ._llcrypt unregistered [ 1912.209777] Key type ._llcrypt registered [ 1912.217731] Key type .llcrypt registered [ 1913.883757] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 11:11:06 (1787843466) [ 1914.213277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1914.225509] alg: No test for adler32 (adler32-zlib) [ 1915.131574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1915.206697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1915.243793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1915.277603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1915.310760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1921.270912] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 11:11:14 (1787843474) [ 1921.425934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1921.455800] LNet: 100148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1921.489142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.552167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1921.590927] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1922.447273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1922.463712] LNet: 100293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.472549] LNetError: 100293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.490580] LNet: Removed LNI 192.168.202.10@tcp [ 1922.507785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1922.555525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1922.574961] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1923.023121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1923.049278] LNetError: 100321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1923.067872] LNet: Removed LNI 192.168.202.10@tcp [ 1923.092983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.130735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1923.151127] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1928.871634] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 11:11:22 (1787843482) [ 1928.976449] LNet: 100906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.992232] LNet: 100906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1929.003159] LNetError: 100906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1929.020907] LNet: Removed LNI 192.168.202.10@tcp [ 1929.671689] Key type .llcrypt unregistered [ 1929.676328] Key type ._llcrypt unregistered [ 1936.574374] Key type ._llcrypt registered [ 1936.580384] Key type .llcrypt registered [ 1936.891688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1936.918939] alg: No test for adler32 (adler32-zlib) [ 1937.841367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1937.907108] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1937.910176] LNet: *** cfs_fail_loc=e003, val=100*** [ 1938.641456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1938.664975] LNet: *** cfs_fail_loc=e003, val=100*** [ 1938.669159] LNet: Skipped 2 previous similar messages [ 1938.679897] Ping reply received with mapping: [ 1938.683794] found metadata reply NI[192.168.202.10@tcp] [ 1938.687664] 0a ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1938.693453] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1938.700808] 5a 5a 5a 5a 5a 5a 5a 5a [ 1938.858759] LNet: 101312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.863526] LNetError: 101312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1938.874969] LNet: Removed LNI 192.168.202.10@tcp [ 1939.524572] Key type .llcrypt unregistered [ 1939.536465] Key type ._llcrypt unregistered [ 1952.968696] Key type ._llcrypt registered [ 1952.971670] Key type .llcrypt registered [ 1954.241896] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 11:11:47 (1787843507) [ 1954.942290] Key type .llcrypt unregistered [ 1954.944070] Key type ._llcrypt unregistered [ 1961.469072] Key type ._llcrypt registered [ 1961.475203] Key type .llcrypt registered [ 1961.882336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1961.918465] alg: No test for adler32 (adler32-zlib) [ 1963.083657] LNetError: 102370:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1963.446362] Key type .llcrypt unregistered [ 1963.449277] Key type ._llcrypt unregistered [ 1968.908569] Key type ._llcrypt registered [ 1968.911340] Key type .llcrypt registered [ 1970.702809] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 11:12:03 (1787843523) [ 1971.198432] Key type .llcrypt unregistered [ 1971.204045] Key type ._llcrypt unregistered [ 1977.561437] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1977.569042] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1980.304712] Key type ._llcrypt registered [ 1980.309847] Key type .llcrypt registered [ 1980.535797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1980.560236] alg: No test for adler32 (adler32-zlib) [ 1981.522297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.619607] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1992.011610] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 11:12:24 (1787843544) [ 1992.098109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.132628] LNet: 104153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.146121] LNetError: 104153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1992.185319] LNet: Removed LNI 10.1.2.3@tcp [ 1992.216832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.996779] Key type .llcrypt unregistered [ 1993.000090] Key type ._llcrypt unregistered [ 1998.888095] Key type ._llcrypt registered [ 1998.889769] Key type .llcrypt registered [ 1999.170369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1999.208262] alg: No test for adler32 (adler32-zlib) [ 2000.126914] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2000.195843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2000.343450] LNet: 104452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.350395] LNetError: 104452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2001.384757] LNet: Removed LNI 192.168.202.10@tcp [ 2002.073593] Key type .llcrypt unregistered [ 2002.077780] Key type ._llcrypt unregistered [ 2013.162761] Key type ._llcrypt registered [ 2013.164695] Key type .llcrypt registered [ 2014.089234] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 11:12:47 (1787843567) [ 2015.013670] Key type .llcrypt unregistered [ 2015.018730] Key type ._llcrypt unregistered [ 2020.272984] Key type ._llcrypt registered [ 2020.274850] Key type .llcrypt registered [ 2020.468457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2020.484957] alg: No test for adler32 (adler32-zlib) [ 2027.346915] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 11:13:00 (1787843580) [ 2027.450932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2027.498592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2027.569643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2027.592947] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2028.303230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2028.354680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2028.392444] LNet: Removed LNI 192.168.202.10@tcp [ 2028.719863] LNet: 106207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2029.149430] Key type .llcrypt unregistered [ 2029.153519] Key type ._llcrypt unregistered [ 2039.607589] Key type ._llcrypt registered [ 2039.611410] Key type .llcrypt registered [ 2040.795131] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 11:13:13 (1787843593) [ 2041.378578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2041.389344] alg: No test for adler32 (adler32-zlib) [ 2042.370146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2042.433903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2042.461677] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2042.961929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2042.991701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.10@tcp [ 2043.008475] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2043.011187] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2043.013840] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2042, 0, 0 [ 2043.017877] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2043.020180] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2043.023165] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2043.025568] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2042, 0, 0 [ 2043.029549] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2043.031478] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2043.033455] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2043.044137] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2042, 0, 2092 [ 2043.048390] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2043.051281] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2043.057467] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2043.060364] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2042, 0, 2092 [ 2043.064435] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2049.311496] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 11:13:22 (1787843602) [ 2049.747161] LNet: 107746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2049.756961] LNetError: 107746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2049.777316] LNet: Removed LNI 192.168.202.10@tcp [ 2050.303464] Key type .llcrypt unregistered [ 2050.310071] Key type ._llcrypt unregistered [ 2056.791771] Key type ._llcrypt registered [ 2056.796640] Key type .llcrypt registered [ 2057.068617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2057.100435] alg: No test for adler32 (adler32-zlib) [ 2057.991407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2058.014770] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2066.050823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 2076.582278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.669886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2076.712568] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 2077.824172] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2078.035433] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2078.049076] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2078.059968] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2080.421434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2080.441893] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2091.373360] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 11:14:04 (1787843644) [ 2091.780935] LNet: 109261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.787636] LNetError: 109261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2091.795967] LNet: Removed LNI 192.168.202.10@tcp [ 2092.833812] LNet: Removed LNI 192.168.202.10@tcp1 [ 2092.842534] LNetError: 109261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2092.855341] LNetError: 109261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2092.880943] LNet: Removed LNI 10.1.2.3@tcp99 [ 2093.643795] Key type .llcrypt unregistered [ 2093.646120] Key type ._llcrypt unregistered [ 2100.239529] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2100.249611] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2102.950153] Key type ._llcrypt registered [ 2102.952542] Key type .llcrypt registered [ 2103.166602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2103.182227] alg: No test for adler32 (adler32-zlib) [ 2104.092646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2104.153828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2104.174695] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2104.878126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2104.907622] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2107.190782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2107.255914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2107.324418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2107.349509] LNetError: 110009:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2107.376916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2107.422369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2108.003522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2108.078682] LNet: 110029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.083707] LNetError: 110029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2108.088554] LNet: Removed LNI 10.1.2.3@tcp [ 2109.160661] LNet: Removed LNI 192.168.202.10@tcp [ 2109.821591] Key type .llcrypt unregistered [ 2109.823753] Key type ._llcrypt unregistered [ 2123.499538] Key type ._llcrypt registered [ 2123.501638] Key type .llcrypt registered [ 2125.188645] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 11:14:37 (1787843677) [ 2125.994406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2126.016345] alg: No test for adler32 (adler32-zlib) [ 2126.961876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.024889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2127.044299] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2127.764298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2127.809859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg210-client@tcp [ 2133.378443] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 11:14:46 (1787843686) [ 2133.619731] LNet: 111564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2133.638292] LNetError: 111564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2133.697743] LNet: Removed LNI 192.168.202.10@tcp [ 2134.359665] Key type .llcrypt unregistered [ 2134.361756] Key type ._llcrypt unregistered [ 2140.932613] Key type ._llcrypt registered [ 2140.935453] Key type .llcrypt registered [ 2141.096860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2141.119857] alg: No test for adler32 (adler32-zlib) [ 2142.015794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2142.035695] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2142.068339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2148.203894] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 11:15:01 (1787843701) [ 2149.713444] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2151.718025] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 11:15:04 (1787843704) [ 2153.440587] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2155.080393] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 11:15:08 (1787843708) [ 2155.188806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2155.202497] LNet: 112982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2155.212439] LNetError: 112982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2155.232657] LNet: Removed LNI 192.168.202.10@tcp [ 2155.259293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.334277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2155.359424] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2156.038299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2156.074612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2162.550936] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 11:15:15 (1787843715) [ 2162.633185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2162.676802] LNet: 113691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2162.689143] LNetError: 113691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2162.704916] LNet: Removed LNI 192.168.202.10@tcp [ 2162.728097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2162.814673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2162.860524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2163.443256] Key type .llcrypt unregistered [ 2163.445123] Key type ._llcrypt unregistered [ 2175.356463] Key type ._llcrypt registered [ 2175.360977] Key type .llcrypt registered [ 2176.968911] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 11:15:29 (1787843729) [ 2177.373683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2177.393367] alg: No test for adler32 (adler32-zlib) [ 2178.363939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2178.421748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2178.442702] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2179.371858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2179.425198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2185.745855] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 11:15:39 (1787843739) [ 2186.094317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2186.122779] LNet: 115258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2186.131811] LNetError: 115258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2186.143528] LNet: Removed LNI 192.168.202.10@tcp [ 2186.170648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.274935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2186.326755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2192.551926] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 11:15:45 (1787843745) [ 2192.729452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2192.761120] LNet: 115836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2192.830202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2192.949889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2193.358819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2193.398314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2193.467267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2193.987575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2194.041543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2194.104790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2194.126643] LNet: 115872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2194.143870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2194.196617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2194.510053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2194.566895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2194.627345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2195.098656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2195.125652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2201.684075] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 11:15:54 (1787843754) [ 2201.824580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2201.851805] LNet: 116469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2201.880736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2201.977940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2202.001076] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2202.681485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2202.726655] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2203.445740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2203.492263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2203.664321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2203.699800] LNetError: 116751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2203.710315] LNet: Removed LNI 192.168.202.10@tcp [ 2203.737320] LNet: Removed LNI 192.168.202.10@tcp2 [ 2203.787585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.861598] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2203.869800] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2203.877763] LNet: 116757:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2204.498847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2204.734698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2204.766535] LNetError: 116794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2204.772739] LNetError: 116794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2204.780636] LNet: Removed LNI 192.168.202.10@tcp [ 2204.790993] LNet: Removed LNI 192.168.202.10@tcp2 [ 2204.807724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2204.888293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2204.912800] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2205.702593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2205.725608] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2206.622763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2206.651440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2206.699322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2206.870602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2206.886557] LNet: 117079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.890775] LNet: 117079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2206.895593] LNetError: 117079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2206.902637] LNetError: 117079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2206.908213] LNet: Removed LNI 192.168.202.10@tcp [ 2207.987291] LNet: Removed LNI 192.168.202.10@tcp2 [ 2208.025863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2208.114830] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2208.123131] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2208.129964] LNet: 117085:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2208.804900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2208.991562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2209.002963] LNet: Removed LNI 192.168.202.10@tcp [ 2210.028352] LNet: Removed LNI 192.168.202.10@tcp2 [ 2210.052172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.189885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2210.225079] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2211.024539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2211.041759] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2211.645703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2211.691458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2211.874659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2211.892626] LNetError: 117401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2211.895822] LNetError: 117401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2211.901680] LNet: Removed LNI 192.168.202.10@tcp [ 2211.909593] LNet: Removed LNI 192.168.202.10@tcp2 [ 2211.923462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2211.970435] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2211.974626] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2211.981600] LNet: 117407:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2212.588202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2212.806789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2213.856379] LNet: Removed LNI 192.168.202.10@tcp [ 2213.872807] LNet: Removed LNI 192.168.202.10@tcp2 [ 2213.918451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2213.999332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2214.019164] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2214.608919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2214.629378] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2215.277176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2215.316473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2215.371754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2215.542718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2215.560182] LNet: 117729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2215.563985] LNet: 117729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2216.608072] LNet: Removed LNI 192.168.202.10@tcp [ 2217.643206] LNet: Removed LNI 192.168.202.10@tcp2 [ 2217.673279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2217.727372] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2217.738479] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2217.750267] LNet: 117735:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2218.225846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2218.494785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2218.527826] LNet: Removed LNI 192.168.202.10@tcp [ 2219.563691] LNet: Removed LNI 192.168.202.10@tcp2 [ 2219.622611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.702582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2219.740274] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2220.459623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2220.482814] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2221.521928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2221.574952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2221.648507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2222.034662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2222.055712] LNetError: 118069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.063297] LNetError: 118069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2222.070634] LNet: Removed LNI 192.168.202.10@tcp [ 2222.085465] LNet: Removed LNI 192.168.202.10@tcp2 [ 2222.137458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.283958] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2222.289866] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2222.309575] LNet: 118075:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2222.866464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2223.179722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2224.224414] LNet: Removed LNI 192.168.202.10@tcp [ 2224.234598] LNet: Removed LNI 192.168.202.10@tcp2 [ 2224.258416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2224.352067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2224.392031] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2225.190854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2225.229105] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2226.147397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2226.224771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2226.273801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2226.335749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2226.673193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2227.744465] LNet: Removed LNI 192.168.202.10@tcp [ 2228.775333] LNet: Removed LNI 192.168.202.10@tcp2 [ 2228.814769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2228.898860] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2228.904120] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2229.528038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2237.957992] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2239.581053] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 11:16:32 (1787843792) [ 2240.078231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2240.134664] LNet: 119195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2240.146925] LNet: 119195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2240.158960] LNetError: 119195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2240.172139] LNetError: 119195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2240.179909] LNet: Removed LNI 192.168.202.10@tcp [ 2241.255519] LNet: Removed LNI 192.168.202.10@tcp2 [ 2241.303501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.736283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2241.763253] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2242.352894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2242.377826] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2243.362486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2243.398325] LNet: Removed LNI 10.1.2.3@tcp [ 2243.407695] LNet: Removed LNI 192.168.202.10@tcp [ 2243.447604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.527858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2243.560393] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2243.567167] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2243.971958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2243.994213] LNet: Removed LNI 192.168.202.10@tcp [ 2244.021873] LNet: Removed LNI 10.1.2.3@tcp [ 2250.385760] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 11:16:43 (1787843803) [ 2250.615785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2250.664670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2250.747971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2250.781045] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2251.557644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2251.597857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2251.640198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2251.671635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2251.746519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2251.773486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2251.801322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2251.828962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2251.859066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2251.888805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2251.935555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2252.100558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2258.281289] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 11:16:51 (1787843811) [ 2258.390418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2259.501079] LNet: Removed LNI 192.168.202.10@tcp [ 2259.559973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2259.686774] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2259.893740] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2259.898958] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2259.901089] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2262.266994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2262.338670] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2263.238883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2263.272089] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2265.649894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2265.703159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2265.816658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.10@tcp -m PUT -e local_timeout -r 1 [ 2265.906688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2265.987424] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2265.997944] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2266.283573] LNetError: 120875:0:(lib-move.c:4801:lnet_parse()) 192.168.202.10@tcp, src 192.168.202.10@tcp: Dropping PUT (error -108 looking up sender) [ 2267.301822] LNet: Removed LNI 10.1.2.3@tcp [ 2268.338202] LNet: Removed LNI 192.168.202.10@tcp [ 2268.971676] Key type .llcrypt unregistered [ 2268.978278] Key type ._llcrypt unregistered [ 2282.504863] Key type ._llcrypt registered [ 2282.506918] Key type .llcrypt registered [ 2283.963344] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 11:17:16 (1787843836) [ 2284.452648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2284.464618] alg: No test for adler32 (adler32-zlib) [ 2285.430986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2285.830888] LNet: 122190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2286.586505] Key type .llcrypt unregistered [ 2286.588314] Key type ._llcrypt unregistered [ 2292.652373] Key type ._llcrypt registered [ 2292.655952] Key type .llcrypt registered [ 2292.885270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2292.911033] alg: No test for adler32 (adler32-zlib) [ 2293.788791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2293.809049] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2301.381799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 2301.659228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2302.621877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.110@tcp [ 2302.996723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2303.037235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2303.069979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2303.072803] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 2303.108628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2303.134627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2303.160924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2303.191949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2303.225587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2303.248876] LNet: 122645:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2303.268914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2303.304643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2303.353197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2303.392860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2303.418803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2303.452611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2304.480155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2304.990849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2305.046646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2305.111661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2305.168395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2305.228680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2305.277532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2305.347081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2305.414691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2305.441857] LNet: 122762:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2305.471716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2305.520798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2305.566604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2305.596678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2305.670064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2305.727484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2306.637315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.110@tcp [ 2306.910312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2306.948687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2307.016946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2307.072534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2307.146449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2307.179509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2307.214715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2307.261592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2307.286883] LNet: 122880:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2307.309826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2307.352951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2307.400977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2307.453943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2307.537552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2307.607758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2308.740354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2309.171295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2309.255634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2309.332481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2309.400297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2309.514865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2309.588495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2309.658261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2309.732855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2309.826659] LNet: 122997:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2309.869833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2309.978789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2310.041445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2310.100551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2310.139906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2310.218936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2311.082657] LNet: 123047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.091799] LNetError: 123047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2311.133170] LNet: Removed LNI 192.168.202.10@tcp [ 2311.662854] Key type .llcrypt unregistered [ 2311.668571] Key type ._llcrypt unregistered [ 2338.260181] Key type ._llcrypt registered [ 2338.263798] Key type .llcrypt registered [ 2340.087085] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 11:18:12 (1787843892) [ 2340.539703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2340.562923] alg: No test for adler32 (adler32-zlib) [ 2341.473668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2341.956513] LNet: 123935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2342.552474] Key type .llcrypt unregistered [ 2342.556677] Key type ._llcrypt unregistered [ 2349.850356] Key type ._llcrypt registered [ 2349.853612] Key type .llcrypt registered [ 2350.167405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2350.189384] alg: No test for adler32 (adler32-zlib) [ 2351.146912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2351.181794] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2358.614921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 2367.362420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2367.385726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2367.399869] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 2368.099216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2368.574368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.110@tcp [ 2368.609931] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 2368.796711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2368.847049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2368.885957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2368.911371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2368.948413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2368.980424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2369.009443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2369.037529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2369.057104] LNet: 124593:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2369.071932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2369.099940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2369.124702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2369.149326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2369.194137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2369.215723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2369.756854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2369.992761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2370.025188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2370.056854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2370.088508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2370.119692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2370.145901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2370.169519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2370.225567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2370.247608] LNet: 124710:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2370.271941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2370.310651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2370.358419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2370.392268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2370.438174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2370.468732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2371.211479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.110@tcp [ 2371.413485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2371.433554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2371.461777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2371.493148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2371.549805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2371.599424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2371.662915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2371.719528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2371.760110] LNet: 124827:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2371.779210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2371.819686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2371.879920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2371.924524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2371.964886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2372.029890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2372.693516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2372.947880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2373.008730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2373.046981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2373.093697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2373.130856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2373.166498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2373.208767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2373.252671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2373.299293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2373.339826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2373.383767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2373.419671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2373.480979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2373.514837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2379.091471] LNet: 125030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2379.122919] LNetError: 125030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2379.152714] LNet: Removed LNI 192.168.202.10@tcp [ 2379.201459] LNet: Removed LNI 192.168.202.10@tcp1 [ 2379.731665] Key type .llcrypt unregistered [ 2379.740542] Key type ._llcrypt unregistered [ 2399.138710] Key type ._llcrypt registered [ 2399.142617] Key type .llcrypt registered [ 2400.679329] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 11:19:13 (1787843953) [ 2401.225740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2401.247355] alg: No test for adler32 (adler32-zlib) [ 2402.252663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2402.310193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2402.345658] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2403.252921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2403.291636] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2404.446264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2404.537368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2405.601191] LNet: Removed LNI 192.168.202.10@tcp [ 2412.014021] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 11:19:25 (1787843965) [ 2413.868708] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2415.401094] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 11:19:28 (1787843968) [ 2423.320781] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 11:19:36 (1787843976) [ 2431.719806] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 11:19:44 (1787843984) [ 2440.194278] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 11:19:53 (1787843993) [ 2448.639194] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 11:20:01 (1787844001) [ 2456.475109] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 11:20:09 (1787844009) [ 2464.484554] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 11:20:17 (1787844017) [ 2472.284993] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 11:20:25 (1787844025) [ 2481.344624] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 11:20:34 (1787844034) [ 2488.508842] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 11:20:41 (1787844041) [ 2495.837267] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 11:20:48 (1787844048) [ 2503.615267] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 11:20:56 (1787844056) [ 2510.550421] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 11:21:03 (1787844063) [ 2517.126817] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 11:21:10 (1787844070) [ 2523.937777] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 11:21:17 (1787844077) [ 2531.179249] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 11:21:24 (1787844084) [ 2537.782076] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 11:21:30 (1787844090) [ 2544.238207] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 11:21:37 (1787844097) [ 2544.417591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2544.463228] LNet: 136730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2544.470702] LNetError: 136730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2544.485708] LNet: Removed LNI 192.168.202.10@tcp2 [ 2544.515208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2544.654183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2544.675326] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2545.723642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2545.743737] LNet: 136872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2545.748400] LNetError: 136872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2545.758797] LNet: Removed LNI 192.168.202.10@tcp [ 2545.772880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2545.813943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2545.827263] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2551.779365] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 11:21:45 (1787844105) [ 2559.491884] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 11:21:52 (1787844112) [ 2566.819623] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 11:21:59 (1787844119) [ 2574.184631] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 11:22:07 (1787844127) [ 2580.739910] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 11:22:13 (1787844133) [ 2589.190500] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 11:22:21 (1787844141) [ 2597.140349] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 11:22:30 (1787844150) [ 2605.064870] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 11:22:38 (1787844158) [ 2614.347687] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 11:22:47 (1787844167) [ 2621.704968] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 11:22:54 (1787844174) [ 2628.123841] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 11:23:01 (1787844181) [ 2635.169761] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 11:23:08 (1787844188) [ 2641.642512] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2393 sec ========== 11:23:14 (1787844194) [ 2642.144243] LNet: 144337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2642.150885] LNetError: 144337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2642.162897] LNet: Removed LNI 192.168.202.10@tcp [ 2642.921654] Key type .llcrypt unregistered [ 2642.923513] Key type ._llcrypt unregistered [ 2655.859437] Key type ._llcrypt registered [ 2655.873794] Key type .llcrypt registered [ 2656.139088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2656.161972] alg: No test for adler32 (adler32-zlib) [ 2657.271300] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 2657.526104] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2659.232418] Key type lgssc registered [ 2660.607160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2729.327416] Lustre: Mounted lustre-client - version 2.17.57_92_gffc8991 [ 2734.810865] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2744.710937] Lustre: Unmounted lustre-client [ 2766.354430] Key type lgssc unregistered [ 2766.607907] LNet: 147299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2766.624704] LNetError: 147299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2766.650467] LNet: Removed LNI 192.168.202.10@tcp [ 2767.583315] Key type .llcrypt unregistered [ 2767.586910] Key type ._llcrypt unregistered