[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 440765568 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002350] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007360] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010162] LSM: Security Framework initializing [ 0.011070] Yama: becoming mindful. [ 0.012046] SELinux: Initializing. [ 0.014076] *** VALIDATE selinux *** [ 0.023756] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028637] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029176] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031143] *** VALIDATE tmpfs *** [ 0.033194] *** VALIDATE proc *** [ 0.034302] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.037295] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039130] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041031] Spectre V2 : User space: Vulnerable [ 0.042014] Speculative Store Bypass: Vulnerable [ 0.045792] debug: unmapping init [mem 0xffffffffb9059000-0xffffffffb9060fff] [ 0.047207] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048746] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049028] ... version: 2 [ 0.050016] ... bit width: 48 [ 0.051011] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053017] ... max period: 00007fffffffffff [ 0.054030] ... fixed-purpose events: 3 [ 0.055011] ... event mask: 000000070000000f [ 0.056369] rcu: Hierarchical SRCU implementation. [ 0.058592] smp: Bringing up secondary CPUs ... [ 0.059639] x86: Booting SMP configuration: [ 0.060033] .... node #0, CPUs: #1 #2 #3 [ 0.064443] smp: Brought up 1 node, 4 CPUs [ 0.066015] smpboot: Max logical packages: 1 [ 0.067017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.164369] node 0 deferred pages initialised in 93ms [ 0.166112] devtmpfs: initialized [ 0.168242] x86/mm: Memory block size: 128MB [ 0.171921] gcov: version magic: 0x41383552 [ 0.172737] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.177069] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.178307] pinctrl core: initialized pinctrl subsystem [ 0.179206] [ 0.179859] ************************************************************* [ 0.180017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181016] ** ** [ 0.182014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.183014] ** ** [ 0.184014] ** This means that this kernel is built to expose internal ** [ 0.185012] ** IOMMU data structures, which may compromise security on ** [ 0.186015] ** your system. ** [ 0.187017] ** ** [ 0.188016] ** If you see this message and you are not debugging the ** [ 0.189028] ** kernel, report this immediately to your vendor! ** [ 0.190016] ** ** [ 0.191014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192018] ************************************************************* [ 0.194079] NET: Registered protocol family 16 [ 0.195471] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.196077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.197070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.198529] cpuidle: using governor menu [ 0.200329] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.203477] PCI: Using configuration type 1 for base access [ 0.205129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.215111] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.219044] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223094] cryptd: max_cpu_qlen set to 1000 [ 0.226239] ACPI: Added _OSI(Module Device) [ 0.227012] ACPI: Added _OSI(Processor Device) [ 0.228015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.229017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.233057] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235409] ACPI: Interpreter enabled [ 0.236069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.237030] ACPI: Using IOAPIC for interrupt routing [ 0.238105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.239433] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.247892] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.251098] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.254027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.257082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.262921] acpiphp: Slot [2] registered [ 0.265186] acpiphp: Slot [5] registered [ 0.267143] acpiphp: Slot [6] registered [ 0.268134] acpiphp: Slot [3] registered [ 0.270102] acpiphp: Slot [4] registered [ 0.271096] acpiphp: Slot [7] registered [ 0.272101] acpiphp: Slot [8] registered [ 0.274106] acpiphp: Slot [9] registered [ 0.275125] acpiphp: Slot [10] registered [ 0.276093] acpiphp: Slot [11] registered [ 0.278082] acpiphp: Slot [12] registered [ 0.279073] acpiphp: Slot [13] registered [ 0.280078] acpiphp: Slot [14] registered [ 0.281063] acpiphp: Slot [15] registered [ 0.282113] acpiphp: Slot [16] registered [ 0.283097] acpiphp: Slot [17] registered [ 0.284068] acpiphp: Slot [18] registered [ 0.286077] acpiphp: Slot [19] registered [ 0.287119] acpiphp: Slot [20] registered [ 0.288132] acpiphp: Slot [21] registered [ 0.290141] acpiphp: Slot [22] registered [ 0.292105] acpiphp: Slot [23] registered [ 0.293107] acpiphp: Slot [24] registered [ 0.295155] acpiphp: Slot [25] registered [ 0.296195] acpiphp: Slot [26] registered [ 0.298157] acpiphp: Slot [27] registered [ 0.300147] acpiphp: Slot [28] registered [ 0.301118] acpiphp: Slot [29] registered [ 0.302086] acpiphp: Slot [30] registered [ 0.303100] acpiphp: Slot [31] registered [ 0.305080] PCI host bridge to bus 0000:00 [ 0.307014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.309036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.311029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.314027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.317032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.320055] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.322293] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.325065] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.328401] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.337017] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.341522] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.344021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.347024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.350022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.352732] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.355653] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.358037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.360468] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.363668] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.371774] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.375014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.379689] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.385020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.389024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.404023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.413598] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.419019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.424015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.435020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.444380] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.446636] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.449291] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.450221] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.452203] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.455201] iommu: Default domain type: Passthrough [ 0.457320] SCSI subsystem initialized [ 0.458098] ACPI: bus type USB registered [ 0.459028] usbcore: registered new interface driver usbfs [ 0.460051] usbcore: registered new interface driver hub [ 0.462083] usbcore: registered new device driver usb [ 0.464215] pps_core: LinuxPPS API ver. 1 registered [ 0.465013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.468064] PTP clock support registered [ 0.471062] EDAC MC: Ver: 3.0.0 [ 0.472268] PCI: Using ACPI for IRQ routing [ 0.475121] NetLabel: Initializing [ 0.477012] NetLabel: domain hash size = 128 [ 0.478011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.481091] NetLabel: unlabeled traffic allowed by default [ 0.484073] vgaarb: loaded [ 0.485613] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.487016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.495000] clocksource: Switched to clocksource kvm-clock [ 0.602386] VFS: Disk quotas dquot_6.6.0 [ 0.604016] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.606499] *** VALIDATE ramfs *** [ 0.607598] *** VALIDATE hugetlbfs *** [ 0.609174] pnp: PnP ACPI init [ 0.612381] pnp: PnP ACPI: found 6 devices [ 0.628185] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.631422] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.633414] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.635270] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.637723] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.639692] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.642395] NET: Registered protocol family 2 [ 0.644954] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.650321] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.653751] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.659072] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.661768] TCP: Hash tables configured (established 65536 bind 65536) [ 0.664561] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.666763] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.669397] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.671915] NET: Registered protocol family 1 [ 0.674303] RPC: Registered named UNIX socket transport module. [ 0.676238] RPC: Registered udp transport module. [ 0.677803] RPC: Registered tcp transport module. [ 0.679557] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.681456] NET: Registered protocol family 44 [ 0.682623] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.684453] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.686072] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.687711] PCI: CLS 0 bytes, default 64 [ 0.689343] Unpacking initramfs... [ 2.132194] debug: unmapping init [mem 0xffff98403cc64000-0xffff98403ffcffff] [ 2.136445] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.138914] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.141363] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.617561] Initialise system trusted keyrings [ 2.619029] Key type blacklist registered [ 2.620834] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.629684] zbud: loaded [ 2.632972] *** VALIDATE nfs *** [ 2.634154] *** VALIDATE nfs4 *** [ 2.635535] pstore: using deflate compression [ 2.638753] Platform Keyring initialized [ 2.752495] NET: Registered protocol family 38 [ 2.754527] Key type asymmetric registered [ 2.756265] Asymmetric key parser 'x509' registered [ 2.758311] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.761615] io scheduler mq-deadline registered [ 2.763607] io scheduler kyber registered [ 2.765600] io scheduler bfq registered [ 2.768144] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.771590] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.774469] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.778193] ACPI: Power Button [PWRF] [ 2.785177] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.793164] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.813870] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.841423] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.870275] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.874914] Non-volatile memory driver v1.3 [ 2.876750] Linux agpgart interface v0.103 [ 2.907864] virtio_blk virtio1: [vda] 146696 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.911019] vda: detected capacity change from 0 to 75108352 [ 2.947150] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.950339] vdb: detected capacity change from 0 to 1073741824 [ 2.979732] libphy: Fixed MDIO Bus: probed [ 2.994264] usbcore: registered new interface driver usbserial_generic [ 2.996845] usbserial: USB Serial support registered for generic [ 2.998566] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.002504] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.004069] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.006955] mousedev: PS/2 mouse device common for all mice [ 3.009795] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.014947] rtc_cmos 00:05: RTC can wake from S4 [ 3.017621] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.021226] rtc_cmos 00:05: registered as rtc0 [ 3.021448] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.023188] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.028891] intel_pstate: CPU model not supported [ 3.032290] hid: raw HID events driver (C) Jiri Kosina [ 3.035067] usbcore: registered new interface driver usbhid [ 3.037469] usbhid: USB HID core driver [ 3.039168] drop_monitor: Initializing network drop monitor service [ 3.041933] Initializing XFRM netlink socket [ 3.044117] NET: Registered protocol family 10 [ 3.047146] Segment Routing with IPv6 [ 3.048758] NET: Registered protocol family 17 [ 3.050942] mpls_gso: MPLS GSO support [ 3.056979] RAS: Correctable Errors collector initialized. [ 3.059116] AVX version of gcm_enc/dec engaged. [ 3.060676] AES CTR mode by8 optimization enabled [ 3.144315] sched_clock: Marking stable (3144292134, 0)->(4053617331, -909325197) [ 3.147720] registered taskstats version 1 [ 3.150446] Loading compiled-in X.509 certificates [ 3.153302] zswap: loaded using pool lzo/zbud [ 3.188582] Key type big_key registered [ 3.203853] Key type encrypted registered [ 3.205289] ima: No TPM chip found, activating TPM-bypass! [ 3.207517] ima: Allocated hash algorithm: sha1 [ 3.209757] ima: No architecture policies found [ 3.214079] evm: Initialising EVM extended attributes: [ 3.217177] evm: security.selinux [ 3.218548] evm: security.ima [ 3.219858] evm: security.capability [ 3.221452] evm: HMAC attrs: 0x1 [ 3.224570] rtc_cmos 00:05: setting system clock to 2026-09-13 11:04:10 UTC (1789297450) [ 3.232625] debug: unmapping init [mem 0xffffffffba003000-0xffffffffba1fffff] [ 3.236380] debug: unmapping init [mem 0xffffffffb8d82000-0xffffffffb9058fff] [ 3.248262] Write protecting the kernel read-only data: 28672k [ 3.256300] debug: unmapping init [mem 0xffffffffb7403000-0xffffffffb75fffff] [ 3.261380] debug: unmapping init [mem 0xffffffffb7d14000-0xffffffffb7dfffff] [ 3.316945] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.336801] systemd[1]: Detected virtualization kvm. [ 3.343723] systemd[1]: Detected architecture x86-64. [ 3.347955] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.431686] systemd[1]: No hostname configured. [ 3.439395] systemd[1]: Set hostname to . [ 3.451359] random: systemd: uninitialized urandom read (16 bytes read) [ 3.460602] systemd[1]: Initializing machine ID from random generator. [ 3.567545] random: ln: uninitialized urandom read (6 bytes read) [ 3.793974] random: systemd: uninitialized urandom read (16 bytes read) [ 3.803430] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.813377] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.820730] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.345984] device-mapper: uevent: version 1.0.3 [ 5.351430] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.684928] random: fast init done [ 7.765794] virtio_net virtio0 ens2: renamed from eth0 [ 8.033736] scsi host0: ata_piix [ 8.083903] scsi host1: ata_piix [ 8.091499] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 8.102143] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 12.735220] random: crng init done [ 12.737622] random: 7 urandom warning(s) missed due to ratelimiting [ 14.522924] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.474230] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.760059] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.811163] SELinux: Disabled at runtime. [ 21.021246] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.058880] systemd[1]: Detected virtualization kvm. [ 21.068717] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.737566] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.750924] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.758451] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.786528] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.796837] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.827445] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.851790] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ 23.099128] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 24.970985] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.404988] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.511792] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.985133] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.126978] EDAC sbridge: Ver: 1.1.2 [ 30.811428] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (8s / no limit)[ 31.522862] NFS: Registering the id_resolver key type [ 31.532423] Key type id_resolver registered [ 31.542624] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg244-client login: [ 106.501740] libcfs: loading out-of-tree module taints kernel. [ 106.797793] Key type ._llcrypt registered [ 106.806026] Key type .llcrypt registered [ 107.583337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 107.595682] alg: No test for adler32 (adler32-zlib) [ 109.048904] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 109.857795] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 111.751433] Key type lgssc registered [ 114.231792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 141.580291] hrtimer: interrupt took 10153590 ns [ 246.185901] Lustre: Mounted lustre-client - version 2.17.57_108_gfed9771 [ 252.799254] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 270.307882] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing check_logdir /tmp/testlogs/ [ 271.839269] Lustre: lustre-OST0000-osc-ffff984083ed0800: disconnect after 23s idle [ 276.165593] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing yml_node [ 281.749379] Lustre: DEBUG MARKER: Client: 2.17.57.108 [ 285.793415] Lustre: DEBUG MARKER: MDS: 2.17.57.108 [ 287.955434] Lustre: DEBUG MARKER: OSS: 2.17.57.108 [ 290.208848] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Sep 13 07:08:55 EDT 2026 [ 306.685898] Lustre: Unmounted lustre-client [ 328.218725] Key type lgssc unregistered [ 328.530840] LNet: 8424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.537490] LNetError: 8424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 328.554631] LNet: Removed LNI 192.168.202.44@tcp [ 329.297163] Key type .llcrypt unregistered [ 329.300654] Key type ._llcrypt unregistered [ 356.408764] Key type ._llcrypt registered [ 356.413754] Key type .llcrypt registered [ 356.663732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 356.682259] alg: No test for adler32 (adler32-zlib) [ 357.639291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 357.696024] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 357.718287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 362.445639] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 362.953583] LNet: 9305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.970326] LNetError: 9305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 363.018512] LNet: Removed LNI 192.168.202.44@tcp [ 364.202080] Key type .llcrypt unregistered [ 364.205122] Key type ._llcrypt unregistered [ 372.386309] Key type ._llcrypt registered [ 372.388367] Key type .llcrypt registered [ 373.808173] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 07:10:19 (1789297819) [ 374.522438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 374.533332] alg: No test for adler32 (adler32-zlib) [ 375.561232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.680346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.721429] LNet: 9711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.752528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.844688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.896783] LNet: 9718:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 382.795800] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 07:10:28 (1789297828) [ 383.055709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.079757] LNet: 10296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.111239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.223446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 389.242134] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 07:10:35 (1789297835) [ 389.617500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.667580] LNet: 10883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.695375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.824405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 398.581899] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 07:10:44 (1789297844) [ 398.672923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.698619] LNet: 11461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.723210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.817653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 404.699254] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 07:10:50 (1789297850) [ 405.012906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.036315] LNet: 12053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 405.051716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.096869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 405.614358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 411.741632] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 07:10:58 (1789297858) [ 412.016350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.058686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.131875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 412.514610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 412.880216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 418.992259] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 07:11:05 (1789297865) [ 419.114824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.140405] LNet: 13238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 419.143836] LNet: 13238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 419.187682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.256664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 426.150382] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 07:11:12 (1789297872) [ 426.420819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.506775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.578836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 426.605656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 426.864587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 426.892562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 427.227990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 427.259698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 427.636741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 427.692801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 428.729205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 428.783511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 429.425453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 429.466686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 430.024292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 430.062126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 430.709704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 430.764678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 431.410730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 431.466905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 432.257854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 432.325114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 433.111915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 433.178765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 433.994912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 434.100883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 434.747300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 434.834965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 441.820250] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 07:11:27 (1789297887) [ 441.943700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.005464] LNet: 14499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 442.012723] LNet: 14499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 442.057194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.205474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 442.277175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 449.516279] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 07:11:35 (1789297895) [ 449.582498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.689802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.833074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 449.887877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 458.457598] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 07:11:44 (1789297904) [ 458.608751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.687665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.784432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 458.828490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 465.852920] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 07:11:51 (1789297911) [ 466.060390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.127199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.278349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 466.342310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 474.321349] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 07:11:59 (1789297919) [ 474.479883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.493401] LNet: 16853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.496973] LNet: 16853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 474.509582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.561262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 474.584809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 482.606623] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 07:12:08 (1789297928) [ 482.875501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.913957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.033624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 483.119774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 490.889701] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 07:12:16 (1789297936) [ 491.166564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.207893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.333319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 491.872784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 498.764480] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 07:12:24 (1789297944) [ 498.865781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.902138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.091761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 499.266562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 505.610857] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 07:12:31 (1789297951) [ 505.851522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.891725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.048753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 506.301375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 512.008178] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 07:12:38 (1789297958) [ 512.202226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.268770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.574728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 512.841857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 519.701543] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 07:12:45 (1789297965) [ 519.799174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.887771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.047396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 520.111362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 528.019472] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 07:12:53 (1789297973) [ 528.117560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.189286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.345466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 528.414967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 535.208380] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 07:13:01 (1789297981) [ 535.473293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.523595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.603524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 542.399842] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 07:13:08 (1789297988) [ 542.553343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.596949] LNet: 22289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 542.610919] LNet: 22289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 542.676615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.770522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 550.146667] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 07:13:15 (1789297995) [ 550.315532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.366410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.428887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 550.783924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 558.114349] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 07:13:24 (1789298004) [ 558.316898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.351420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.442401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 558.491130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 566.096338] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 07:13:31 (1789298011) [ 566.335841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.453101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.578359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 566.629299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 574.012331] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 07:13:39 (1789298019) [ 574.208620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.248344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.323455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 574.370148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 580.235916] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 07:13:46 (1789298026) [ 580.490346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.536670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.617535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 580.647078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 586.887482] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 07:13:52 (1789298032) [ 587.124589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.194954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.270068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 587.338162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 587.349388] LNetError: 25825:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 587.379702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 587.407182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 593.283308] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 07:13:59 (1789298039) [ 593.397072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.447272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.564526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.fEavvRFpGq [ 593.625577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.fEavvRFpGq [ 593.670754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.fEavvRFpGq [ 599.955117] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 07:14:05 (1789298045) [ 600.125825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.185699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.347609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 600.397260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 600.460594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 607.319650] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 07:14:13 (1789298053) [ 607.415311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.463400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.555571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 607.585484] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 608.983934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 610.261748] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 612.179679] Key type lgssc registered [ 613.815735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 617.683234] LNetError: 27584:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.44@tcp99 has no available nets [ 620.285561] Key type lgssc unregistered [ 620.612152] LNetError: 28691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 620.632498] LNet: Removed LNI 192.168.202.44@tcp [ 621.124357] Key type .llcrypt unregistered [ 621.126875] Key type ._llcrypt unregistered [ 628.338916] Key type ._llcrypt registered [ 628.341984] Key type .llcrypt registered [ 629.893305] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 07:14:35 (1789298075) [ 630.343324] Key type .llcrypt unregistered [ 630.344855] Key type ._llcrypt unregistered [ 637.416742] Key type ._llcrypt registered [ 637.419796] Key type .llcrypt registered [ 637.607359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 637.633621] alg: No test for adler32 (adler32-zlib) [ 638.539543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.614242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 638.651981] LNet: 29601:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 638.664754] LNet: 29601:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 638.674413] LNet: 29601:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 638.701637] LNet: Message forwarding will be enabled in 2s [ 640.735610] LNet: Message forwarding enabled [ 641.244831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 641.265885] LNet: Message forwarding will stop in 2s to 6s [ 644.836750] LNet: Message forwarding disabled [ 645.951138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 645.991789] LNet: 29691:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 646.004684] LNet: 29691:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 646.014481] LNet: 29691:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 646.036375] LNet: Message forwarding will be enabled in 2s [ 646.292155] LNet: 29704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.068829] Key type .llcrypt unregistered [ 647.070828] Key type ._llcrypt unregistered [ 658.588539] Key type ._llcrypt registered [ 658.592709] Key type .llcrypt registered [ 660.197791] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 07:15:05 (1789298105) [ 660.676799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 660.700892] alg: No test for adler32 (adler32-zlib) [ 661.683128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.799361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 668.571094] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 07:15:14 (1789298114) [ 668.855639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.885228] LNet: 31104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.906284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.982720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 669.004723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 669.027324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 669.052786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 669.077750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 669.107250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 669.126232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 669.158449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 669.190463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 669.222652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 669.248049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 669.272832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 669.306544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 669.334576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 669.362523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 669.386690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 669.414607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 669.451254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 669.485532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 669.505256] LNetError: 31147:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 674.752670] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 07:15:20 (1789298120) [ 674.855438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.881769] LNet: 31712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.923755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.985141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 680.964550] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 07:15:26 (1789298126) [ 683.267882] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg244-server [ 685.356355] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 07:15:30 (1789298130) [ 685.441744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.473356] LNet: 32641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.501237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.594291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 685.648335] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 686.502479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[45-49]@tcp [ 686.520991] LNet: 32782:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 686.581641] LNetError: 32648:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.45@tcp is being used as a gateway but routing feature is not turned on [ 686.603640] LNetError: 32648:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.45@tcp has gone from up to down [ 694.289078] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 07:15:39 (1789298139) [ 696.227890] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg244-server [ 698.007307] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 07:15:43 (1789298143) [ 698.347369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.374370] LNet: 33718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.386154] LNetError: 33718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.406426] LNet: Removed LNI 192.168.202.44@tcp [ 698.436563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.522603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 698.551390] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 699.483099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[45-50/2]@tcp [ 699.522416] LNet: 33836:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 699.529177] LNet: 33836:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 699.558312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[45-50/2]@tcp [ 699.571245] LNetError: 33722:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.49@tcp is being used as a gateway but routing feature is not turned on [ 699.583310] LNetError: 33722:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 699.601778] LNetError: 33722:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.49@tcp has gone from up to down [ 699.607249] LNetError: 33722:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 706.965990] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 07:15:52 (1789298152) [ 707.085619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.138479] LNet: 34422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.145191] LNetError: 34422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.178877] LNet: Removed LNI 192.168.202.44@tcp [ 707.229219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.366521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 707.432138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.469078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.560931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 707.630498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.678832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.757987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 707.860653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.936433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.007553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 708.113485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.150062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.223480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 708.342596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.401992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.520492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 708.645854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.694733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.777797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 708.905896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.952683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.021397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 709.120993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.156745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.214700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 709.313745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.402492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.497289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 709.630264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.697284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.756967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 709.842153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.885426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.972569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 717.079775] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 07:16:02 (1789298162) [ 718.968305] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg244-server [ 721.140292] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 07:16:06 (1789298166) [ 723.264380] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg244-server [ 725.938751] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 07:16:11 (1789298171) [ 726.113844] LNet: 35823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.132902] LNet: 35823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 727.092217] Key type .llcrypt unregistered [ 727.094201] Key type ._llcrypt unregistered [ 735.440512] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 735.724446] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 735.756241] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 735.772341] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 737.638987] Key type ._llcrypt registered [ 737.647968] Key type .llcrypt registered [ 738.039745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 738.083886] alg: No test for adler32 (adler32-zlib) [ 739.126349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.230870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 739.302836] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 740.193960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 740.214053] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 742.726217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 742.764439] LNet: Removed LNI 10.1.2.3@tcp [ 742.993486] LNet: 36541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.006174] LNetError: 36541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.024072] LNet: Removed LNI 192.168.202.44@tcp [ 743.943466] Key type .llcrypt unregistered [ 743.945114] Key type ._llcrypt unregistered [ 757.023984] Key type ._llcrypt registered [ 757.026424] Key type .llcrypt registered [ 758.662891] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 07:16:44 (1789298204) [ 759.145347] Key type .llcrypt unregistered [ 759.147498] Key type ._llcrypt unregistered [ 766.230032] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 766.382200] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 766.392678] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 766.404437] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 768.998258] Key type ._llcrypt registered [ 769.000576] Key type .llcrypt registered [ 769.151813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 769.162913] alg: No test for adler32 (adler32-zlib) [ 770.119690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.184486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 770.216519] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 770.801394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 770.818041] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 772.920763] LNet: 38030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.927397] LNetError: 38030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 772.935219] LNet: Removed LNI 10.1.2.3@tcp [ 772.949770] LNet: Removed LNI 192.168.202.44@tcp [ 773.582255] Key type .llcrypt unregistered [ 773.588841] Key type ._llcrypt unregistered [ 783.330168] Key type ._llcrypt registered [ 783.333448] Key type .llcrypt registered [ 784.631305] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 07:17:10 (1789298230) [ 785.141305] Key type .llcrypt unregistered [ 785.143411] Key type ._llcrypt unregistered [ 791.418929] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 791.506495] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 791.519464] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 791.533989] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 794.134253] Key type ._llcrypt registered [ 794.136204] Key type .llcrypt registered [ 794.322833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 794.344607] alg: No test for adler32 (adler32-zlib) [ 795.278398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.329190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 795.345384] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 797.381426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 797.400366] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 799.364921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 799.398705] LNet: Removed LNI 10.1.2.3@tcp [ 799.414043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 799.449889] LNet: Removed LNI 10.1.2.31@tcp [ 799.465108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 799.484211] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.180403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 800.196411] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 800.998758] LNet: 39942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.011224] LNetError: 39942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 801.014046] LNet: Removed LNI 10.1.2.31@tcp [ 801.030747] LNet: Removed LNI 10.1.2.3@tcp [ 801.543135] Key type .llcrypt unregistered [ 801.546104] Key type ._llcrypt unregistered [ 812.678633] Key type ._llcrypt registered [ 812.687326] Key type .llcrypt registered [ 814.159819] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 07:17:39 (1789298259) [ 814.550881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 814.586451] alg: No test for adler32 (adler32-zlib) [ 815.649083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.714768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 815.745593] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 816.741914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.768960] LNet: 40892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.775692] LNetError: 40892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.794134] LNet: Removed LNI 192.168.202.44@tcp [ 816.822284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.917624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.44@tcp [ 816.975692] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 817.921586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.44@tcp [ 817.956723] LNet: Removed LNI 192.168.202.44@tcp [ 818.056609] LNet: 41027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.714663] Key type .llcrypt unregistered [ 818.717964] Key type ._llcrypt unregistered [ 830.652752] Key type ._llcrypt registered [ 830.654882] Key type .llcrypt registered [ 831.923886] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 07:17:57 (1789298277) [ 833.755888] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg244-server [ 835.600499] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 07:18:01 (1789298281) [ 836.150545] Key type .llcrypt unregistered [ 836.151563] Key type ._llcrypt unregistered [ 843.026359] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 843.139844] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 843.147156] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 843.155470] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 845.862410] Key type ._llcrypt registered [ 845.865648] Key type .llcrypt registered [ 846.142536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 846.161730] alg: No test for adler32 (adler32-zlib) [ 847.045972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.115888] LNet: Added LNI 192.168.202.44@tcp1 [8/256/0/180] [ 847.976030] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 849.989442] LNet: 42855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.994724] LNetError: 42855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.001552] LNet: Removed LNI 192.168.202.44@tcp1 [ 850.013610] LNet: Removed LNI 10.1.2.3@tcp2 [ 850.571304] Key type .llcrypt unregistered [ 850.574062] Key type ._llcrypt unregistered [ 862.251385] Key type ._llcrypt registered [ 862.253062] Key type .llcrypt registered [ 863.700679] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 07:18:29 (1789298309) [ 864.351862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 864.362739] alg: No test for adler32 (adler32-zlib) [ 865.358970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.422379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 865.474127] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 866.244409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.258414] LNet: 43821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.263956] LNetError: 43821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.278913] LNet: Removed LNI 192.168.202.44@tcp [ 866.297703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.360918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 866.398048] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 872.814627] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 07:18:38 (1789298318) [ 873.034902] LNet: 44409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.050143] LNetError: 44409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.086536] LNet: Removed LNI 192.168.202.44@tcp [ 873.627140] Key type .llcrypt unregistered [ 873.628837] Key type ._llcrypt unregistered [ 880.866926] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 881.066546] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 881.083389] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 881.102314] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 883.823809] Key type ._llcrypt registered [ 883.825976] Key type .llcrypt registered [ 884.076471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 884.098828] alg: No test for adler32 (adler32-zlib) [ 885.053584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.106355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 885.137093] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 886.009382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 886.039911] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 888.307510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 888.325935] LNet: 45151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.332374] LNetError: 45151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 888.342704] LNet: Removed LNI 10.1.2.3@tcp [ 888.354062] LNet: Removed LNI 192.168.202.44@tcp [ 888.380552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.454929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 888.487678] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 888.494444] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 888.996689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.020721] LNet: 45178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.028479] LNetError: 45178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 889.034946] LNetError: 45178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 889.041641] LNet: Removed LNI 10.1.2.3@tcp [ 889.047913] LNet: Removed LNI 192.168.202.44@tcp [ 889.089827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.143459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 889.194098] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 889.209883] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 889.687701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 889.702407] LNet: Added LNI 192.168.202.44@tcp2 [8/256/0/180] [ 890.772567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 890.793946] LNet: 45340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.818725] LNetError: 45340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 890.827867] LNetError: 45340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 890.832113] LNet: Removed LNI 10.1.2.3@tcp [ 890.833490] LNet: Removed LNI 192.168.202.44@tcp [ 890.860355] LNet: Removed LNI 192.168.202.44@tcp2 [ 890.904300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.957377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 890.982189] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 890.987417] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 890.993890] LNet: Added LNI 192.168.202.44@tcp2 [8/256/0/180] [ 898.705436] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 07:19:04 (1789298344) [ 899.065273] LNet: 45940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.080221] LNetError: 45940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 899.086401] LNetError: 45940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 899.093740] LNet: Removed LNI 10.1.2.3@tcp [ 899.095439] LNet: Removed LNI 192.168.202.44@tcp [ 899.124481] LNet: Removed LNI 192.168.202.44@tcp2 [ 899.619597] Key type .llcrypt unregistered [ 899.623203] Key type ._llcrypt unregistered [ 905.920258] Key type ._llcrypt registered [ 905.923409] Key type .llcrypt registered [ 906.143233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 906.170913] alg: No test for adler32 (adler32-zlib) [ 907.096938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.148808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 907.182066] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 907.788654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.806620] LNet: 46333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.811300] LNetError: 46333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.824642] LNet: Removed LNI 192.168.202.44@tcp [ 907.834991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.870333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 907.883508] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 908.325586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.345251] LNet: 46363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.350414] LNetError: 46363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.363935] LNet: Removed LNI 192.168.202.44@tcp [ 908.381288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.434248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 908.459636] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 909.109740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.161720] LNet: Removed LNI 192.168.202.44@tcp [ 909.179790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.253767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 909.289084] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 910.007489] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 910.168173] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 910.173792] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 910.177154] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 912.695459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.727479] LNet: 46472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.737363] LNet: 46472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 912.743766] LNetError: 46472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.748691] LNetError: 46472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 912.761413] LNet: Removed LNI 192.168.202.44@tcp [ 912.830910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.958770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 912.996635] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 919.974822] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 07:19:26 (1789298366) [ 920.399588] LNet: 47069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.410278] LNetError: 47069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.437216] LNet: Removed LNI 192.168.202.44@tcp [ 920.966827] Key type .llcrypt unregistered [ 920.970645] Key type ._llcrypt unregistered [ 928.127528] Key type ._llcrypt registered [ 928.134562] Key type .llcrypt registered [ 928.380558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 928.396566] alg: No test for adler32 (adler32-zlib) [ 929.395902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.487161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 929.516910] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 930.324741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.347774] LNet: 47464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.351857] LNetError: 47464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.372545] LNet: Removed LNI 192.168.202.44@tcp [ 930.391475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.480866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 930.527756] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 938.399640] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 07:19:43 (1789298383) [ 938.535819] LNet: 48047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.546175] LNetError: 48047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.575366] LNet: Removed LNI 192.168.202.44@tcp [ 939.189767] Key type .llcrypt unregistered [ 939.191384] Key type ._llcrypt unregistered [ 945.543850] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 945.748184] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 945.761150] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 945.765640] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 948.501923] Key type ._llcrypt registered [ 948.506233] Key type .llcrypt registered [ 948.807503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 948.831674] alg: No test for adler32 (adler32-zlib) [ 949.849710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.930492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 949.961253] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 950.890625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.922228] LNet: 48502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.926913] LNetError: 48502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.947994] LNet: Removed LNI 192.168.202.44@tcp [ 950.978525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.052885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 951.097606] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 958.723794] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 07:20:04 (1789298404) [ 959.009248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 959.046092] LNet: 49100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.070740] LNetError: 49100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.087425] LNet: Removed LNI 192.168.202.44@tcp [ 959.119574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.309735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 968.959528] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 07:20:13 (1789298413) [ 969.168766] LNet: 49676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.593736] Key type .llcrypt unregistered [ 970.597140] Key type ._llcrypt unregistered [ 978.636293] Key type ._llcrypt registered [ 978.639899] Key type .llcrypt registered [ 978.870828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 978.884387] alg: No test for adler32 (adler32-zlib) [ 979.798877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.880669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 979.915582] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 981.074664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.114412] LNet: 50079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.120842] LNetError: 50079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.140808] LNet: Removed LNI 192.168.202.44@tcp [ 981.159689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.265532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 981.302239] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 989.926520] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 07:20:35 (1789298435) [ 989.994051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 990.023062] LNet: 50665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.030296] LNetError: 50665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.062483] LNet: Removed LNI 192.168.202.44@tcp [ 990.081961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.152908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 995.798034] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 07:20:41 (1789298441) [ 997.203473] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 998.900993] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 07:20:44 (1789298444) [ 1000.265507] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1002.004765] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 07:20:47 (1789298447) [ 1003.645498] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1005.303829] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 07:20:51 (1789298451) [ 1006.817167] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1008.577913] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 07:20:54 (1789298454) [ 1010.059318] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1011.774389] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 07:20:57 (1789298457) [ 1011.963755] LNet: 52722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.501634] Key type .llcrypt unregistered [ 1012.503653] Key type ._llcrypt unregistered [ 1018.520883] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1018.680851] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1018.708254] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1018.715780] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1021.390598] Key type ._llcrypt registered [ 1021.392775] Key type .llcrypt registered [ 1021.527619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1021.541782] alg: No test for adler32 (adler32-zlib) [ 1022.494952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1022.566764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1022.587550] LNet: Added LNI 192.168.202.44@tcp [8/512/0/180] [ 1022.594723] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1022.597541] LNet: 53043:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1022.715496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1022.741933] LNet: 53062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.750859] LNetError: 53062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1022.756152] LNet: Removed LNI 10.1.2.3@tcp [ 1022.758413] LNetError: 53062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.770549] LNet: Removed LNI 192.168.202.44@tcp [ 1022.792953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1022.851518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1022.885402] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1022.963870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1022.986613] LNet: 53086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.994897] LNetError: 53086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.007995] LNet: Removed LNI 192.168.202.44@tcp [ 1023.036926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.090576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1023.114220] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1023.120773] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1023.123169] LNet: 53094:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1023.208103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1023.226228] LNet: 53112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.234209] LNetError: 53112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1023.238914] LNet: Removed LNI 10.1.2.3@tcp [ 1023.241255] LNetError: 53112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.253240] LNet: Removed LNI 192.168.202.44@tcp [ 1023.274884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.342030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1023.365644] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1023.370879] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1023.373320] LNet: 53120:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1023.583943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1023.610336] LNet: 53146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.615955] LNetError: 53146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1023.621593] LNet: Removed LNI 10.1.2.3@tcp [ 1023.625140] LNetError: 53146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.640247] LNet: Removed LNI 192.168.202.44@tcp [ 1023.668973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.749717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1023.780126] LNet: Added LNI 192.168.202.44@tcp [8/256/0/182] [ 1023.787072] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1023.987274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1024.030455] LNet: 53174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.042504] LNetError: 53174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1024.052713] LNet: Removed LNI 10.1.2.3@tcp [ 1024.058991] LNetError: 53174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.079717] LNet: Removed LNI 192.168.202.44@tcp [ 1024.156690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.212482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1024.233530] LNet: Added LNI 192.168.202.44@tcp [8/256/0/182] [ 1024.238267] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1024.241655] LNet: 53182:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1024.318423] LNet: Removed LNI 10.1.2.3@tcp [ 1024.330315] LNet: Removed LNI 192.168.202.44@tcp [ 1025.035178] Key type .llcrypt unregistered [ 1025.036618] Key type ._llcrypt unregistered [ 1036.980490] Key type ._llcrypt registered [ 1036.983888] Key type .llcrypt registered [ 1038.608436] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 07:21:24 (1789298484) [ 1040.195108] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1042.607256] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 07:21:27 (1789298487) [ 1043.088610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1043.105403] alg: No test for adler32 (adler32-zlib) [ 1044.187552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1044.285965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1044.312899] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1045.689461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.44 [ 1045.712754] LNet: 54443:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1045.727367] LNet: Removed LNI 192.168.202.44@tcp [ 1054.124710] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 07:21:39 (1789298499) [ 1054.609189] LNet: 55013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.619170] LNetError: 55013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.655698] LNet: Removed LNI 192.168.202.44@tcp [ 1055.552637] Key type .llcrypt unregistered [ 1055.560728] Key type ._llcrypt unregistered [ 1063.032729] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1063.239727] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1066.097794] Key type ._llcrypt registered [ 1066.100390] Key type .llcrypt registered [ 1066.242553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1066.259848] alg: No test for adler32 (adler32-zlib) [ 1067.779181] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1075.581768] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 07:22:01 (1789298521) [ 1075.764174] LNet: 55919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.770792] LNetError: 55919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1075.806046] LNet: Removed LNI 10.1.2.3@tcp [ 1076.551256] Key type .llcrypt unregistered [ 1076.554957] Key type ._llcrypt unregistered [ 1083.214712] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1083.525264] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1083.530672] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1083.533488] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1086.316511] Key type ._llcrypt registered [ 1086.318429] Key type .llcrypt registered [ 1086.456591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1086.464646] alg: No test for adler32 (adler32-zlib) [ 1087.393544] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1094.574655] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 07:22:20 (1789298540) [ 1096.257796] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1097.694853] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 07:22:23 (1789298543) [ 1097.971951] LNet: 57109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.983371] LNetError: 57109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1098.002818] LNet: Removed LNI 10.1.2.3@tcp [ 1098.640554] Key type .llcrypt unregistered [ 1098.642241] Key type ._llcrypt unregistered [ 1104.924773] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1105.152695] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1105.156234] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1105.160926] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1107.892133] Key type ._llcrypt registered [ 1107.894858] Key type .llcrypt registered [ 1108.122767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1108.137550] alg: No test for adler32 (adler32-zlib) [ 1109.108217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1109.182386] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1119.063095] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 07:22:44 (1789298564) [ 1119.424240] LNet: 58335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1119.433944] LNetError: 58335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1119.455109] LNet: Removed LNI 10.1.2.3@tcp [ 1120.149202] Key type .llcrypt unregistered [ 1120.151349] Key type ._llcrypt unregistered [ 1126.810351] Key type ._llcrypt registered [ 1126.811871] Key type .llcrypt registered [ 1127.084328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1127.105919] alg: No test for adler32 (adler32-zlib) [ 1128.110303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1128.161976] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1136.397680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.144@tcp [ 1137.084383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_interrupt -r 1 [ 1137.123791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_interrupt -r 1 [ 1137.150514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1137.170908] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1137.352571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1137.377408] LNet: 1 local NIs in recovery (showing 1): 192.168.202.44@tcp [ 1137.378212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1137.434402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1137.476876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1137.512172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1137.567858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1137.602405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1137.681147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1137.740542] LNet: 58771:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1137.779539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1137.833870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1137.920380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1137.985864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1138.060139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1138.097992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1138.129487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1138.729952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_dropped -r 1 [ 1138.758972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_dropped -r 1 [ 1138.788990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1138.820579] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1139.074502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1139.098934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1139.136722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1139.162818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1139.202518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1139.243887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1139.282931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1139.307337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1139.327198] LNet: 58880:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1139.342741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1139.378340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1139.421446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1139.453765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1139.490227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1139.524679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1139.554197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1140.004787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_aborted -r 1 [ 1140.039916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_aborted -r 1 [ 1140.068854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1140.086546] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1140.091702] LNet: Skipped 1 previous similar message [ 1140.306732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.330746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1140.354189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1140.376679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1140.403661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.430535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1140.453903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1140.479858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1140.493169] LNet: 58989:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1140.509451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1140.530875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1140.556229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1140.581197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.608734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1140.631944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1140.666540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1141.101674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_no_route -r 1 [ 1141.131946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_no_route -r 1 [ 1141.188275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1141.454731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1141.489140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1141.530549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1141.574818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1141.602427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1141.625085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1141.647069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1141.687454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1141.724766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1141.765732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1141.787494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1141.811414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1141.842773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1141.895286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1141.921631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1142.559289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_timeout -r 1 [ 1142.606142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_timeout -r 1 [ 1142.665793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1142.706515] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1142.717323] LNet: Skipped 2 previous similar messages [ 1143.020948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1143.049201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1143.083692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1143.110334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1143.148297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1143.172517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1143.204681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1143.232129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1143.250557] LNet: 59208:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1143.257392] LNet: 59208:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1143.272889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1143.309920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1143.353000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1143.380949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1143.427126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1143.465522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1143.506419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1144.000662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_error -r 1 [ 1144.030208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_error -r 1 [ 1144.067405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1144.327842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1144.358711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1144.383501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1144.416667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1144.447763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1144.495629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1144.526267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1144.579223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1144.619463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1144.672441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1144.709212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1144.750461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1144.780083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1144.805054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1144.834393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1145.181212] LNet: 59358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.193377] LNetError: 59358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1146.221876] LNet: Removed LNI 192.168.202.44@tcp [ 1146.830469] Key type .llcrypt unregistered [ 1146.832327] Key type ._llcrypt unregistered [ 1163.166258] Key type ._llcrypt registered [ 1163.168582] Key type .llcrypt registered [ 1164.237292] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 07:23:30 (1789298610) [ 1165.137263] Key type .llcrypt unregistered [ 1165.139135] Key type ._llcrypt unregistered [ 1171.216329] Key type ._llcrypt registered [ 1171.218950] Key type .llcrypt registered [ 1171.391145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1171.403692] alg: No test for adler32 (adler32-zlib) [ 1172.387390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1172.430043] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1179.865225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.144@tcp [ 1190.664432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1190.748869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1190.785333] LNet: Added LNI 192.168.202.44@tcp1 [8/256/0/180] [ 1192.325930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_interrupt -r 1 [ 1192.366896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp1 -m GET -e local_interrupt -r 1 [ 1192.401332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_interrupt -r 1 [ 1192.471767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -m GET -e local_interrupt -r 1 [ 1192.564897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp -m GET -e local_interrupt -r 1 [ 1192.618219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp1 -m GET -e local_interrupt -r 1 [ 1192.705476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -m GET -e local_interrupt -r 1 [ 1192.770240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -m GET -e local_interrupt -r 1 [ 1192.837629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1192.859989] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1192.865691] LNet: 1 local NIs in recovery (showing 1): 192.168.202.44@tcp [ 1193.148456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1193.231605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1193.291576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1193.323737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1193.347511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1193.370249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1193.393545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1193.416504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1193.429951] LNet: 60822:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1193.442893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1193.467554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1193.498955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1193.526806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1193.549577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.572993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1193.603432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1194.202301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_dropped -r 1 [ 1194.241612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp1 -m GET -e local_dropped -r 1 [ 1194.275418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_dropped -r 1 [ 1194.331826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -m GET -e local_dropped -r 1 [ 1194.383446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp -m GET -e local_dropped -r 1 [ 1194.450909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp1 -m GET -e local_dropped -r 1 [ 1194.506339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -m GET -e local_dropped -r 1 [ 1194.566237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -m GET -e local_dropped -r 1 [ 1194.610249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1194.632213] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1194.638289] LNet: Skipped 2 previous similar messages [ 1194.941630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1194.982412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1195.022383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1195.096280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1195.171922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1195.247492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1195.323660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1195.356278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1195.378815] LNet: 60943:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1195.400959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1195.431465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1195.468898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1195.505578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1195.556613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1195.597930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1195.647121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1196.213932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_aborted -r 1 [ 1196.242598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp1 -m GET -e local_aborted -r 1 [ 1196.277314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_aborted -r 1 [ 1196.320821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -m GET -e local_aborted -r 1 [ 1196.363213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp -m GET -e local_aborted -r 1 [ 1196.399481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp1 -m GET -e local_aborted -r 1 [ 1196.443681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -m GET -e local_aborted -r 1 [ 1196.488946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -m GET -e local_aborted -r 1 [ 1196.542992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1196.564378] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1196.569250] LNet: Skipped 2 previous similar messages [ 1196.851144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1196.890538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1196.948825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1197.036174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1197.093994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1197.135968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1197.198659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1197.263869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1197.280382] LNet: 61065:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1197.299492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1197.377379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1197.428877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1197.460248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1197.495694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1197.552196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1197.596915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1198.186552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_no_route -r 1 [ 1198.232071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp1 -m GET -e local_no_route -r 1 [ 1198.270168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_no_route -r 1 [ 1198.309494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -m GET -e local_no_route -r 1 [ 1198.348877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp -m GET -e local_no_route -r 1 [ 1198.373871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp1 -m GET -e local_no_route -r 1 [ 1198.405155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -m GET -e local_no_route -r 1 [ 1198.427299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -m GET -e local_no_route -r 1 [ 1198.449770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1198.653993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1198.676924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1198.706332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1198.732884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1198.767905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1198.803810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1198.859637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1198.899564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1198.944151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1198.985929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1199.032728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1199.093697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1199.144952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1199.193530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1199.239833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1199.787401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_timeout -r 1 [ 1199.836493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp1 -m GET -e local_timeout -r 1 [ 1199.884790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_timeout -r 1 [ 1199.936617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -m GET -e local_timeout -r 1 [ 1199.979756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp -m GET -e local_timeout -r 1 [ 1200.012640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp1 -m GET -e local_timeout -r 1 [ 1200.046285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -m GET -e local_timeout -r 1 [ 1200.088457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -m GET -e local_timeout -r 1 [ 1200.125779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1200.147387] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1200.151651] LNet: Skipped 5 previous similar messages [ 1200.404898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1200.439336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1200.464700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1200.486837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1200.528464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1200.558507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1200.616550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1200.667022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1200.692098] LNet: 61307:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1200.701383] LNet: 61307:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1200.728856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1200.775334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1200.819018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1200.858228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1200.889675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1200.918441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1200.961591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1201.609976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_error -r 1 [ 1201.637103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp1 -m GET -e local_error -r 1 [ 1201.687115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_error -r 1 [ 1201.740281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -m GET -e local_error -r 1 [ 1201.777776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp -m GET -e local_error -r 1 [ 1201.832383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp1 -m GET -e local_error -r 1 [ 1201.940314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -m GET -e local_error -r 1 [ 1201.971838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -m GET -e local_error -r 1 [ 1202.008046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1202.315890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1202.341545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1202.372706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1202.397270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1202.421953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1202.446728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1202.487757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1202.519402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1202.542888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1202.572218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1202.597941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1202.627480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1202.654693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1202.684697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1202.721719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1208.999909] LNet: 61505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.005860] LNetError: 61505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.014654] LNet: Removed LNI 192.168.202.44@tcp [ 1209.024798] LNet: Removed LNI 192.168.202.44@tcp1 [ 1209.488714] Key type .llcrypt unregistered [ 1209.491345] Key type ._llcrypt unregistered [ 1227.459227] Key type ._llcrypt registered [ 1227.460724] Key type .llcrypt registered [ 1228.577555] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 07:24:34 (1789298674) [ 1229.549619] Key type .llcrypt unregistered [ 1229.551391] Key type ._llcrypt unregistered [ 1236.228477] Key type ._llcrypt registered [ 1236.231558] Key type .llcrypt registered [ 1236.475945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1236.506475] alg: No test for adler32 (adler32-zlib) [ 1237.422678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1237.454499] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1244.013913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.144@tcp [ 1244.462519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e remote_dropped -r 1 [ 1244.487602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e remote_dropped -r 1 [ 1244.512982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1244.527444] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1244.699275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1244.728329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1244.752151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1244.790956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1244.855703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1244.885315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1244.913220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1244.935268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1244.952546] LNet: 62749:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1244.987153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1245.008467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1245.033784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.054366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.085880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.127729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1245.156818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1245.571797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e remote_error -r 1 [ 1245.642693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e remote_error -r 1 [ 1245.675593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1245.685760] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1245.892096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1245.923857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1245.961480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1246.016975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1246.081576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1246.135867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1246.167754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1246.191503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1246.224667] LNet: 62858:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1246.250980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1246.282066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1246.313110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1246.341343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1246.376606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1246.411562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1246.447814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1246.974462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e remote_timeout -r 1 [ 1246.998681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e remote_timeout -r 1 [ 1247.025340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1247.039565] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1247.251696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1247.297331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1247.337989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1247.373218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1247.425543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1247.478989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1247.544547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.605293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.639488] LNet: 62967:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1247.661247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1247.756977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1247.820474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1247.881121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1247.917087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1247.967751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1248.016496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1248.323321] LNet: 63009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.326787] LNetError: 63009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1248.335305] LNet: Removed LNI 192.168.202.44@tcp [ 1248.697567] Key type .llcrypt unregistered [ 1248.700404] Key type ._llcrypt unregistered [ 1265.182187] Key type ._llcrypt registered [ 1265.186080] Key type .llcrypt registered [ 1266.477774] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 07:25:12 (1789298712) [ 1267.131753] Key type .llcrypt unregistered [ 1267.134266] Key type ._llcrypt unregistered [ 1272.817920] Key type ._llcrypt registered [ 1272.820299] Key type .llcrypt registered [ 1272.947690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1272.958726] alg: No test for adler32 (adler32-zlib) [ 1273.901220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1273.938353] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1281.386466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.144@tcp [ 1292.897452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1292.977787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1293.013176] LNet: Added LNI 192.168.202.44@tcp1 [8/256/0/180] [ 1294.857705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e remote_dropped -r 1 [ 1294.895504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp1 -m GET -e remote_dropped -r 1 [ 1294.937418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e remote_dropped -r 1 [ 1294.967568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -m GET -e remote_dropped -r 1 [ 1295.010730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp -m GET -e remote_dropped -r 1 [ 1295.056311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp1 -m GET -e remote_dropped -r 1 [ 1295.083214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -m GET -e remote_dropped -r 1 [ 1295.121184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -m GET -e remote_dropped -r 1 [ 1295.169749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1295.185417] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1295.189926] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.144@tcp [ 1295.480776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1295.522804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1295.558576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1295.615971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1295.667845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1295.712586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1295.771883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1295.823099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1295.862438] LNet: 64471:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1295.890531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1295.924769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1295.963297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1296.021481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1296.069707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.111674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1296.187782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1296.242603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.946868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e remote_error -r 1 [ 1297.015598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp1 -m GET -e remote_error -r 1 [ 1297.075395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e remote_error -r 1 [ 1297.164991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -m GET -e remote_error -r 1 [ 1297.266276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp -m GET -e remote_error -r 1 [ 1297.318951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp1 -m GET -e remote_error -r 1 [ 1297.396707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -m GET -e remote_error -r 1 [ 1297.442560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -m GET -e remote_error -r 1 [ 1297.513509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1297.548570] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1297.552413] LNet: Skipped 2 previous similar messages [ 1297.928563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1297.968133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1298.015325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1298.081927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1298.136914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1298.193979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1298.256051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1298.292714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1298.345652] LNet: 64594:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1298.366703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1298.433716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1298.492412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1298.549877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1298.611528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1298.648578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1298.706928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1298.750509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1299.479750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e remote_timeout -r 1 [ 1299.540706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp1 -m GET -e remote_timeout -r 1 [ 1299.605850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e remote_timeout -r 1 [ 1299.694839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -m GET -e remote_timeout -r 1 [ 1299.750421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp -m GET -e remote_timeout -r 1 [ 1299.817715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp1 -m GET -e remote_timeout -r 1 [ 1299.865843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -m GET -e remote_timeout -r 1 [ 1299.911797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -m GET -e remote_timeout -r 1 [ 1299.962724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.144@tcp [ 1299.989587] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1299.992518] LNet: Skipped 1 previous similar message [ 1300.209573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1300.246188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1300.287648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1300.332485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1300.371457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1300.403055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1300.427305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1300.466479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1300.503703] LNet: 64717:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1300.527459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1300.571830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1300.606733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1300.631957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1300.664530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1300.686886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1300.709954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1300.742367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.574325] LNet: 64795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1305.583868] LNetError: 64795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1305.590948] LNet: Removed LNI 192.168.202.44@tcp [ 1305.611260] LNet: Removed LNI 192.168.202.44@tcp1 [ 1305.963912] Key type .llcrypt unregistered [ 1305.967331] Key type ._llcrypt unregistered [ 1323.084570] Key type ._llcrypt registered [ 1323.087079] Key type .llcrypt registered [ 1324.414494] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 07:26:10 (1789298770) [ 1325.662221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1327.277459] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 07:26:13 (1789298773) [ 1328.136532] Key type .llcrypt unregistered [ 1328.138885] Key type ._llcrypt unregistered [ 1333.985976] Key type ._llcrypt registered [ 1333.988329] Key type .llcrypt registered [ 1334.197263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1334.222838] alg: No test for adler32 (adler32-zlib) [ 1335.111900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1335.148405] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1341.621541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.144@tcp [ 1342.154823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e network_timeout -r 1 [ 1342.192645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e network_timeout -r 1 [ 1342.223515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.144@tcp [ 1342.238829] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1342.304173] LNet: 1 local NIs in recovery (showing 1): 192.168.202.44@tcp [ 1342.442949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1342.467755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1342.498255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1342.522232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1342.549358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1342.574674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1342.606684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1342.640441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1342.661796] LNet: 66333:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1342.675857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1342.705456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1342.732484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1342.761688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1342.786683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1342.814331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1343.086230] LNet: 66372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1343.094977] LNetError: 66372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.117715] LNet: Removed LNI 192.168.202.44@tcp [ 1343.447620] Key type .llcrypt unregistered [ 1343.449363] Key type ._llcrypt unregistered [ 1360.284604] Key type ._llcrypt registered [ 1360.286502] Key type .llcrypt registered [ 1360.515734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1360.537779] alg: No test for adler32 (adler32-zlib) [ 1361.506256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1361.524804] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1367.924067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.144@tcp [ 1374.619458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.647722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1374.670533] LNet: Added LNI 192.168.202.44@tcp1 [8/256/0/180] [ 1375.626316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e network_timeout -r 1 [ 1375.655153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp1 -m GET -e network_timeout -r 1 [ 1375.684899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e network_timeout -r 1 [ 1375.713777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -m GET -e network_timeout -r 1 [ 1375.733520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp -m GET -e network_timeout -r 1 [ 1375.756916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp1 -m GET -e network_timeout -r 1 [ 1375.784514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -m GET -e network_timeout -r 1 [ 1375.814712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -m GET -e network_timeout -r 1 [ 1375.848321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.144@tcp [ 1375.871499] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1375.913974] LNet: 1 local NIs in recovery (showing 1): 192.168.202.44@tcp [ 1375.920219] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.144@tcp [ 1376.097889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1376.132946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1376.154433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1376.175980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1376.198827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1376.225636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1376.255617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1376.272876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1376.282864] LNet: 67244:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1376.295629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1376.323572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1376.351146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1376.388735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1376.423715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1376.455794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1380.441197] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1380.835988] LNet: 67318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1380.846542] LNetError: 67318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1381.923789] LNet: Removed LNI 192.168.202.44@tcp [ 1381.926852] LNetError: 67318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1381.937788] LNet: Removed LNI 192.168.202.44@tcp1 [ 1382.428665] Key type .llcrypt unregistered [ 1382.432304] Key type ._llcrypt unregistered [ 1399.531795] Key type ._llcrypt registered [ 1399.533621] Key type .llcrypt registered [ 1400.708640] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 07:27:26 (1789298846) [ 1401.185143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1401.198633] alg: No test for adler32 (adler32-zlib) [ 1402.141885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1402.186346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1402.227534] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1402.975416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1402.992878] LNet: Added LNI 192.168.202.44@tcp1 [8/256/0/180] [ 1403.922215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 1404.002447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1404.087467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1404.160379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1404.226950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1404.263637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1404.321708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.44@tcp [ 1404.351862] LNetError: 68415:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.44@tcp to recovery [ 1405.217955] LNet: 1 local NIs in recovery (showing 1): 192.168.202.44@tcp [ 1405.218206] LNet: There was an unexpected network error while writing to 192.168.202.44: rc = -22 [ 1407.327400] LNet: There was an unexpected network error while writing to 192.168.202.44: rc = -22 [ 1408.007834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1411.433055] LNet: There was an unexpected network error while writing to 192.168.202.44: rc = -22 [ 1411.684293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1411.722652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1411.792048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1411.815357] LNet: 68522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.823348] LNetError: 68522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.831173] LNet: Removed LNI 192.168.202.44@tcp [ 1412.837695] LNetError: 68522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.868637] LNet: Removed LNI 192.168.202.44@tcp1 [ 1412.910350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1413.042233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1413.090814] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1414.071899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1414.100545] LNet: Added LNI 192.168.202.44@tcp1 [8/256/0/180] [ 1414.963923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 1415.015495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1415.078892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1415.127446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1415.181791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1415.224046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1415.268813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1415.327610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.44@tcp [ 1415.345336] LNetError: 68795:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.44@tcp to recovery [ 1416.032298] LNet: There was an unexpected network error while writing to 192.168.202.44: rc = -22 [ 1419.159886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1422.181583] LNet: There was an unexpected network error while writing to 192.168.202.44: rc = -22 [ 1422.187069] LNet: Skipped 1 previous similar message [ 1427.400896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1427.431048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1427.464985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1434.949763] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 07:28:00 (1789298880) [ 1435.214221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1435.247687] LNet: 69513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1435.253942] LNetError: 69513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1435.260989] LNet: Removed LNI 192.168.202.44@tcp [ 1435.270856] LNet: Removed LNI 192.168.202.44@tcp1 [ 1435.313166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.377618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1435.402974] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1436.161389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1436.190652] LNet: Added LNI 192.168.202.44@tcp1 [8/256/0/180] [ 1437.189336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 1437.293414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1437.325473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1437.372616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1437.410639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1437.462639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1437.509670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.44@tcp --health 0 [ 1437.642425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1438.437021] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.44@tcp [ 1438.437103] LNet: There was an unexpected network error while writing to 192.168.202.44: rc = -22 [ 1438.455258] LNet: Skipped 1 previous similar message [ 1453.646796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1453.880638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1453.943774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1453.978253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.44@tcp --health 500 [ 1454.103182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1460.113348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1460.217700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1460.255790] LNet: 69996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1460.266560] LNetError: 69996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1460.273260] LNetError: 69996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1460.301516] LNet: Removed LNI 192.168.202.44@tcp [ 1460.318150] LNet: Removed LNI 192.168.202.44@tcp1 [ 1460.355271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1460.434144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1460.478101] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1461.219571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1461.257965] LNet: Added LNI 192.168.202.44@tcp1 [8/256/0/180] [ 1462.149152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 1462.184853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1462.238784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1462.261680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1462.290816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1462.326865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1462.351863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1462.382983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.44@tcp --health 0 [ 1463.456622] LNet: There was an unexpected network error while writing to 192.168.202.44: rc = -22 [ 1463.459529] LNet: Skipped 3 previous similar messages [ 1463.725414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1474.445795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1474.533058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1474.589417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1482.620606] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 07:28:48 (1789298928) [ 1482.934694] LNet: 71043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.936946] LNetError: 71043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1482.939607] LNetError: 71043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1482.944242] LNet: Removed LNI 192.168.202.44@tcp [ 1482.950593] LNet: Removed LNI 192.168.202.44@tcp1 [ 1483.276641] Key type .llcrypt unregistered [ 1483.278419] Key type ._llcrypt unregistered [ 1489.306166] Key type ._llcrypt registered [ 1489.309616] Key type .llcrypt registered [ 1489.442682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1489.453296] alg: No test for adler32 (adler32-zlib) [ 1490.398596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1490.438866] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1500.356591] Lustre: DEBUG MARKER: Initial discovery [ 1500.382367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.144@tcp [ 1502.186820] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1502.816868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1502.860888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.144@tcp [ 1502.881848] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1502.926504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1502.974057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.144@tcp -r 1 -m GET -l 3 [ 1503.036107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.144@tcp -r 1 -m GET -e local_error [ 1503.090657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.144@tcp -r 1 -m PUT -l 6 [ 1503.785597] LNet: 1 local NIs in recovery (showing 1): 192.168.202.44@tcp [ 1504.499106] Lustre: DEBUG MARKER: Force oleg244-client.virtnet to discover 192.168.202.144@tcp (in background) [ 1504.581086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.144@tcp [ 1506.021965] Lustre: DEBUG MARKER: Force oleg244-server to discover 192.168.202.44@tcp [ 1507.697599] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1508.654070] Lustre: DEBUG MARKER: Wait for 71796 [ 1510.418296] Lustre: DEBUG MARKER: Finished wait on 71796 [ 1510.557715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1510.606527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1510.607245] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1510.830408] LNet: 72253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1510.834648] LNetError: 72253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.846450] LNet: Removed LNI 192.168.202.44@tcp [ 1512.360645] Key type .llcrypt unregistered [ 1512.362618] Key type ._llcrypt unregistered [ 1528.032560] Key type ._llcrypt registered [ 1528.034599] Key type .llcrypt registered [ 1529.054376] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 07:29:35 (1789298975) [ 1529.627085] Key type .llcrypt unregistered [ 1529.628594] Key type ._llcrypt unregistered [ 1535.956405] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1536.078140] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1536.082655] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1536.086414] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1537.481541] Key type ._llcrypt registered [ 1537.484201] Key type .llcrypt registered [ 1537.617747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1537.642679] alg: No test for adler32 (adler32-zlib) [ 1538.558868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.608591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1538.631652] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1539.294471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1539.304321] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1547.736172] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 07:29:53 (1789298993) [ 1547.950637] LNet: 74315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1547.961687] LNetError: 74315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1547.971657] LNet: Removed LNI 10.1.2.3@tcp [ 1547.988831] LNet: Removed LNI 192.168.202.44@tcp [ 1548.554644] Key type .llcrypt unregistered [ 1548.557737] Key type ._llcrypt unregistered [ 1554.037292] Key type ._llcrypt registered [ 1554.039335] Key type .llcrypt registered [ 1554.188030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1554.221225] alg: No test for adler32 (adler32-zlib) [ 1555.123777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.168856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1555.193420] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1555.841530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1555.866133] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1556.589638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.44@tcp [ 1557.663141] LNetError: Unexpected error -22 connecting to 192.168.202.44@tcp at host 192.168.202.44:988 [ 1561.995800] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 07:30:08 (1789299008) [ 1562.173768] LNet: 75438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1562.183798] LNetError: 75438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1562.194674] LNet: Removed LNI 10.1.2.3@tcp [ 1562.227107] LNet: Removed LNI 192.168.202.44@tcp [ 1562.986793] Key type .llcrypt unregistered [ 1562.989736] Key type ._llcrypt unregistered [ 1569.077870] Key type ._llcrypt registered [ 1569.089995] Key type .llcrypt registered [ 1569.352513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1569.378901] alg: No test for adler32 (adler32-zlib) [ 1570.254588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.303908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1570.319884] LNet: Added LNI 192.168.202.44@tcp1 [8/256/0/180] [ 1570.927128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1570.954131] LNet: Added LNI 192.168.202.44@tcp2 [8/256/0/180] [ 1571.695858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.44@tcp1 --nid 192.168.202.44@tcp2 [ 1575.619730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.44@tcp1 192.168.202.44@tcp2 [ 1575.636363] LNetError: 76275:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.44@tcp1 to 192.168.202.44@tcp2 - Route table may be misconfigured [ 1575.652136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.44@tcp2 192.168.202.44@tcp1 [ 1581.623710] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 07:30:27 (1789299027) [ 1581.920728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.952902] LNet: 76849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1581.968333] LNetError: 76849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1583.007720] LNet: Removed LNI 192.168.202.44@tcp1 [ 1583.012871] LNetError: 76849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1583.049437] LNet: Removed LNI 192.168.202.44@tcp2 [ 1583.120439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.209544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1583.229760] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1584.006412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1584.023775] LNet: Added LNI 192.168.202.44@tcp1 [8/256/0/180] [ 1585.026841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 1585.117967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.44@tcp 192.168.202.44@tcp [ 1585.181722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.44@tcp1 192.168.202.44@tcp1 [ 1585.251469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.44@tcp -d 192.168.202.44@tcp -e network_timeout [ 1585.339684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -e network_timeout [ 1585.382714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -e network_timeout [ 1585.426352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -e network_timeout [ 1585.469806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 1585.498390] LNet: There was an unexpected network error while writing to 192.168.202.44: rc = -22 [ 1585.524803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1585.632273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1585.727355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1586.271281] LNet: 1 local NIs in recovery (showing 1): 192.168.202.44@tcp [ 1592.807580] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 07:30:38 (1789299038) [ 1593.041766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.085690] LNet: 77710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.101452] LNetError: 77710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1593.122227] LNet: Removed LNI 192.168.202.44@tcp [ 1594.156773] LNet: Removed LNI 192.168.202.44@tcp1 [ 1594.249256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.399812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1594.740241] LNet: 77748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1595.175507] Key type .llcrypt unregistered [ 1595.177419] Key type ._llcrypt unregistered [ 1609.346987] Key type ._llcrypt registered [ 1609.348938] Key type .llcrypt registered [ 1610.629570] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 07:30:56 (1789299056) [ 1610.989359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1611.018978] alg: No test for adler32 (adler32-zlib) [ 1611.900518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.196940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1615.220572] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 07:31:00 (1789299060) [ 1615.363325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.379845] LNet: 78843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1615.398362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.435783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1615.456545] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1615.849465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1615.864214] LNet: Added LNI 192.168.202.44@tcp1 [8/256/0/180] [ 1616.760241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 1616.791377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp1 [ 1616.828132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp1 [ 1623.807344] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 07:31:09 (1789299069) [ 1626.054803] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg244-server [ 1627.949537] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 07:31:13 (1789299073) [ 1629.735445] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg244-server [ 1631.356769] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 07:31:17 (1789299077) [ 1633.263792] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg244-server [ 1635.277401] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 07:31:20 (1789299080) [ 1636.566844] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg244-server [ 1638.027376] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 07:31:24 (1789299084) [ 1639.930662] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg244-server [ 1641.600133] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 07:31:27 (1789299087) [ 1643.767411] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg244-server [ 1645.501132] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 07:31:31 (1789299091) [ 1647.512786] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg244-server [ 1649.412452] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 07:31:35 (1789299095) [ 1651.684422] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg244-server [ 1654.155738] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 07:31:39 (1789299099) [ 1656.274666] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg244-server [ 1658.318571] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 07:31:44 (1789299104) [ 1658.519600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.571370] LNet: 82716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.588444] LNetError: 82716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.600075] LNet: Removed LNI 192.168.202.44@tcp [ 1659.624318] LNet: Removed LNI 192.168.202.44@tcp1 [ 1659.681121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.794363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1659.833072] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1660.418959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1660.500172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 1660.625898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.651981] LNet: 82868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1660.657696] LNetError: 82868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1660.660926] LNetError: 82868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1661.756701] LNet: Removed LNI 192.168.202.44@tcp [ 1661.773657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.803424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1661.821347] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1662.519700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 1662.662309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.682293] LNet: 83013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.688858] LNetError: 83013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1663.720445] LNet: Removed LNI 192.168.202.44@tcp [ 1663.758837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.832577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1663.862872] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1664.563309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1664.683987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 1664.984665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1665.014549] LNetError: 83161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1665.038056] LNet: Removed LNI 192.168.202.44@tcp [ 1665.054218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.106493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1665.130243] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1665.859233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 1666.009207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.058382] LNet: Removed LNI 192.168.202.44@tcp [ 1667.092189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.153894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1667.180447] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1667.964468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1668.035466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 1668.201603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1668.225251] LNet: 83453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.232768] LNet: 83453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1669.289194] LNet: Removed LNI 192.168.202.44@tcp [ 1669.350865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.411171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1669.428892] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1670.226937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 1670.431891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1670.462789] LNetError: 83598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1670.485929] LNetError: 83598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1670.522226] LNet: Removed LNI 192.168.202.44@tcp [ 1670.555754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.609458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1670.652423] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1671.356558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1671.437419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 1671.691073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1672.747130] LNet: Removed LNI 192.168.202.44@tcp [ 1672.805217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.875534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1672.911884] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1673.803600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 1674.197396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.311457] LNet: Removed LNI 192.168.202.44@tcp [ 1675.370289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.478495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1675.551684] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1676.510281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1676.614497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 1676.910919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1676.955817] LNet: 84038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.964462] LNet: 84038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1677.996474] LNet: Removed LNI 192.168.202.44@tcp [ 1678.042874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.159593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1678.203053] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1678.823747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 1679.125368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1679.171043] LNetError: 84184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1679.179059] LNetError: 84184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1680.234591] LNet: Removed LNI 192.168.202.44@tcp [ 1680.263696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1680.323951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1680.345271] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1680.982674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1681.033994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 1681.303634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1682.409588] LNet: Removed LNI 192.168.202.44@tcp [ 1682.424312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.514400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1682.534861] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1683.391915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 1684.715834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.799704] LNet: Removed LNI 192.168.202.44@tcp [ 1685.854479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.953737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1686.030036] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1687.054566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1687.128693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.165212] LNet: Removed LNI 192.168.202.44@tcp [ 1687.196432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.302899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1687.332593] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1688.358464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1688.496389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.531763] LNet: Removed LNI 192.168.202.44@tcp [ 1688.564475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.625904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1688.713986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.761873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.822930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1688.911389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.955818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.034537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1689.155075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.192775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.241934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1689.275074] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1689.997568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.021298] LNet: Removed LNI 192.168.202.44@tcp [ 1690.046382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.107954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1696.935205] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 07:32:23 (1789299143) [ 1697.269556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1697.311084] LNet: 85506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.317859] LNet: 85506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1697.359937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.408718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1697.435663] LNet: Added LNI 192.168.202.44@tcp231 [8/256/0/180] [ 1698.387483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.407794] LNetError: 85644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1698.414975] LNetError: 85644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1698.427545] LNet: Removed LNI 192.168.202.44@tcp231 [ 1698.444442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.533642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1698.558379] LNet: Added LNI 192.168.202.44@tcp231 [8/256/0/180] [ 1698.564882] LNet: 85652:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1699.148130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1699.199364] LNet: Removed LNI 192.168.202.44@tcp231 [ 1699.222584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1699.251786] LNet: Added LNI 192.168.202.44@tcp231 [8/256/0/180] [ 1700.376670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.418535] LNet: Removed LNI 192.168.202.44@tcp231 [ 1700.445578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.508047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1700.550258] LNet: Added LNI 192.168.202.44@tcp231 [8/256/0/0] [ 1700.556147] LNet: 85805:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1701.085173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1701.136853] LNet: Removed LNI 192.168.202.44@tcp231 [ 1701.162856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1701.207837] LNet: Added LNI 192.168.202.44@tcp231 [8/256/0/0] [ 1702.308850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.352000] LNet: Removed LNI 192.168.202.44@tcp231 [ 1702.391534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.499081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1702.533267] LNet: Added LNI 192.168.202.44@tcp231 [8/256/0/60] [ 1702.538904] LNet: 85956:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1703.032886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1703.056785] LNet: Removed LNI 192.168.202.44@tcp231 [ 1703.079937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1703.105532] LNet: Added LNI 192.168.202.44@tcp231 [8/256/0/60] [ 1704.116970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.147748] LNet: Removed LNI 192.168.202.44@tcp231 [ 1704.169392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.265908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1704.294742] LNet: Added LNI 192.168.202.44@tcp231 [8/256/0/180] [ 1704.840214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1704.889827] LNet: Removed LNI 192.168.202.44@tcp231 [ 1704.917380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1704.954243] LNet: Added LNI 192.168.202.44@tcp231 [8/256/0/180] [ 1711.905847] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 07:32:37 (1789299157) [ 1712.205656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.249634] LNet: Removed LNI 192.168.202.44@tcp231 [ 1712.288428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.353273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1712.370750] LNet: Added LNI 192.168.202.44@tcp232 [8/256/0/180] [ 1712.945423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1712.985309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1713.020898] LNet: Removed LNI 192.168.202.44@tcp232 [ 1713.043136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1713.065851] LNet: Added LNI 192.168.202.44@tcp232 [8/256/0/180] [ 1713.716862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1713.744191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1713.771683] LNet: Removed LNI 192.168.202.44@tcp232 [ 1713.782683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1713.799603] LNet: Added LNI 192.168.202.44@tcp232 [8/256/0/180] [ 1714.454681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1714.519096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1714.577276] LNet: Removed LNI 192.168.202.44@tcp232 [ 1714.611940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1714.643091] LNet: Added LNI 192.168.202.44@tcp232 [8/256/0/180] [ 1715.302713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1715.333917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1715.374612] LNet: Removed LNI 192.168.202.44@tcp232 [ 1715.400324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1721.555683] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 07:32:47 (1789299167) [ 1722.642849] Key type .llcrypt unregistered [ 1722.662788] Key type ._llcrypt unregistered [ 1730.113784] Key type ._llcrypt registered [ 1730.116275] Key type .llcrypt registered [ 1730.444688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1730.472412] alg: No test for adler32 (adler32-zlib) [ 1731.495640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1731.570162] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1741.587214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.144@tcp [ 1751.715716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.762993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1751.787874] LNet: Added LNI 192.168.202.44@tcp1 [8/256/0/180] [ 1753.368375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_interrupt -r 2 [ 1753.443683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp1 -m GET -e local_interrupt -r 2 [ 1753.483477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_interrupt -r 2 [ 1753.533043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -m GET -e local_interrupt -r 2 [ 1753.617120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp -m GET -e local_interrupt -r 2 [ 1753.670463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp1 -m GET -e local_interrupt -r 2 [ 1753.732841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -m GET -e local_interrupt -r 2 [ 1753.810151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -m GET -e local_interrupt -r 2 [ 1753.892437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.44@tcp 192.168.202.144@tcp [ 1753.920290] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1753.927452] LNet: 1 local NIs in recovery (showing 1): 192.168.202.44@tcp [ 1754.268283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1754.299168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1754.333538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1754.372469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1754.417756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1754.467547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1754.548436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1754.638269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1754.669309] LNet: 88606:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1754.710702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1754.783321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1754.840405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1754.888336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1754.955506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1755.024641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1755.799346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_dropped -r 2 [ 1755.870302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp1 -m GET -e local_dropped -r 2 [ 1755.945246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_dropped -r 2 [ 1756.009988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -m GET -e local_dropped -r 2 [ 1756.075353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp -m GET -e local_dropped -r 2 [ 1756.122262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp1 -m GET -e local_dropped -r 2 [ 1756.175731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -m GET -e local_dropped -r 2 [ 1756.225390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -m GET -e local_dropped -r 2 [ 1756.295633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.44@tcp 192.168.202.144@tcp [ 1756.336758] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1756.726901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1756.763322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1756.806942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1756.845318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1756.915856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1756.988376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1757.023896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1757.057769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1757.075556] LNet: 88727:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1757.090139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1757.121466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1757.145151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1757.172219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1757.210352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1757.252330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1757.719855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_aborted -r 2 [ 1757.751320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp1 -m GET -e local_aborted -r 2 [ 1757.789593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_aborted -r 2 [ 1757.820628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -m GET -e local_aborted -r 2 [ 1757.852426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp -m GET -e local_aborted -r 2 [ 1757.890906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp1 -m GET -e local_aborted -r 2 [ 1757.919309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -m GET -e local_aborted -r 2 [ 1757.949371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -m GET -e local_aborted -r 2 [ 1757.993486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.44@tcp 192.168.202.144@tcp [ 1758.013373] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1758.288908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1758.328721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1758.366879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1758.404787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1758.442210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1758.489897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1758.524781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1758.579123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1758.617770] LNet: 88847:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1758.635774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1758.666855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1758.722928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1758.749368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1758.790565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1758.825906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1759.398697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_no_route -r 2 [ 1759.438940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp1 -m GET -e local_no_route -r 2 [ 1759.480222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_no_route -r 2 [ 1759.516934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -m GET -e local_no_route -r 2 [ 1759.549840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp -m GET -e local_no_route -r 2 [ 1759.578587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp1 -m GET -e local_no_route -r 2 [ 1759.630174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -m GET -e local_no_route -r 2 [ 1759.695166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -m GET -e local_no_route -r 2 [ 1759.755687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.44@tcp 192.168.202.144@tcp [ 1760.120688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1760.163257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1760.208645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1760.255782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1760.293131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1760.320375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1760.354427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1760.384297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1760.420670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1760.458549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1760.507158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1760.528211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1760.553735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1760.587420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1761.167975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e local_timeout -r 2 [ 1761.231788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp1 -m GET -e local_timeout -r 2 [ 1761.296837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e local_timeout -r 2 [ 1761.334454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -m GET -e local_timeout -r 2 [ 1761.388917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp -m GET -e local_timeout -r 2 [ 1761.456706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp1 -m GET -e local_timeout -r 2 [ 1761.496669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -m GET -e local_timeout -r 2 [ 1761.530574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -m GET -e local_timeout -r 2 [ 1761.575858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.44@tcp 192.168.202.144@tcp [ 1761.601286] LNet: There was an unexpected network error while writing to 192.168.202.144: rc = -22 [ 1761.609931] LNet: Skipped 1 previous similar message [ 1761.910439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1761.966894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1762.038293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1762.087225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1762.132599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1762.181305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1762.224271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1762.262367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1762.285837] LNet: 89088:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1762.291232] LNet: 89088:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1762.322327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1762.378941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1762.417914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1762.450482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1762.472959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1762.510560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1763.039814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp -m GET -e remote_dropped -r 2 [ 1763.098929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.144@tcp1 -m GET -e remote_dropped -r 2 [ 1763.148484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp -m GET -e remote_dropped -r 2 [ 1763.182930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.44@tcp -d 192.168.202.44@tcp1 -m GET -e remote_dropped -r 2 [ 1763.226914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp -m GET -e remote_dropped -r 2 [ 1763.276963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.144@tcp1 -m GET -e remote_dropped -r 2 [ 1763.312676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp -m GET -e remote_dropped -r 2 [ 1763.343349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.44@tcp1 -d 192.168.202.44@tcp1 -m GET -e remote_dropped -r 2 [ 1763.375348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.44@tcp 192.168.202.144@tcp [ 1763.395756] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.144@tcp [ 1763.639277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1763.679528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1763.719127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1763.760650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1763.795700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1763.826877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1763.859370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1763.893479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1763.950665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1764.012123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1764.087042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1764.143684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1764.193964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1764.241824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1769.478652] LNet: 89282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1769.483269] LNetError: 89282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1769.488830] LNet: Removed LNI 192.168.202.44@tcp [ 1769.499671] LNet: Removed LNI 192.168.202.44@tcp1 [ 1769.973793] Key type .llcrypt unregistered [ 1769.976252] Key type ._llcrypt unregistered [ 1787.704243] Key type ._llcrypt registered [ 1787.706670] Key type .llcrypt registered [ 1789.208840] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 07:33:54 (1789299234) [ 1789.908824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1789.931660] alg: No test for adler32 (adler32-zlib) [ 1790.869599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1790.915281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1790.940600] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1791.706132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1791.739410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.44@tcp [ 1791.760459] LNetError: 90257:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.44@tcp to recovery [ 1791.834773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.44@tcp [ 1791.933630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1799.344834] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 07:34:05 (1789299245) [ 1799.510992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.568977] LNet: 90846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.573948] LNetError: 90846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.582352] LNet: Removed LNI 192.168.202.44@tcp [ 1799.598070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.651983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1799.675529] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1800.306945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1800.322959] LNet: Added LNI 192.168.202.44@tcp1 [8/256/0/180] [ 1801.101811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.44@tcp [ 1801.134648] LNetError: 91113:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.44@tcp to recovery [ 1801.696158] LNet: 1 local NIs in recovery (showing 1): 192.168.202.44@tcp [ 1801.698080] LNet: There was an unexpected network error while writing to 192.168.202.44: rc = -22 [ 1802.380968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.44@tcp [ 1810.594764] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 07:34:16 (1789299256) [ 1812.618491] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg244-server [ 1814.882236] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 07:34:20 (1789299260) [ 1817.221730] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg244-server [ 1818.960689] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 07:34:24 (1789299264) [ 1820.927201] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg244-server [ 1822.409469] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 07:34:28 (1789299268) [ 1822.605439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1822.673161] LNet: 92744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.684258] LNetError: 92744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.708662] LNet: Removed LNI 192.168.202.44@tcp [ 1822.743026] LNet: Removed LNI 192.168.202.44@tcp1 [ 1822.770619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1822.834766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1822.850582] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1823.685320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1823.742427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1823.774808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1823.881021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1823.920777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1823.988458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1831.329664] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 07:34:37 (1789299277) [ 1831.667264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1831.709726] LNet: 93475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1831.727994] LNetError: 93475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.736199] LNetError: 93475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1831.756390] LNet: Removed LNI 192.168.202.44@tcp [ 1831.777274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1831.825710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1831.841536] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1838.689373] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 07:34:44 (1789299284) [ 1838.941932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1839.012707] LNet: 94179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.018908] LNetError: 94179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1839.037492] LNet: Removed LNI 192.168.202.44@tcp [ 1839.069915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1839.128577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1839.150718] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1846.421320] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 07:34:52 (1789299292) [ 1848.099740] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1849.881246] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 07:34:55 (1789299295) [ 1850.430503] LNet: 95213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1850.435576] LNetError: 95213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1850.449854] LNet: Removed LNI 192.168.202.44@tcp [ 1851.255316] Key type .llcrypt unregistered [ 1851.257965] Key type ._llcrypt unregistered [ 1857.387046] Key type ._llcrypt registered [ 1857.388820] Key type .llcrypt registered [ 1857.616333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1857.646331] alg: No test for adler32 (adler32-zlib) [ 1858.581102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1858.607246] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1866.570526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.144@tcp [ 1872.570172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.144@tcp [ 1872.856908] LNet: 95628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.868066] LNetError: 95628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.897303] LNet: Removed LNI 192.168.202.44@tcp [ 1873.274709] Key type .llcrypt unregistered [ 1873.276733] Key type ._llcrypt unregistered [ 1883.983561] Key type ._llcrypt registered [ 1883.986702] Key type .llcrypt registered [ 1885.492656] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1887.276165] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1889.439724] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 07:35:35 (1789299335) [ 1891.624774] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1893.811410] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 07:35:39 (1789299339) [ 1896.269914] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg244-server [ 1898.438309] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 07:35:43 (1789299343) [ 1900.244366] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg244-server [ 1901.838539] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 07:35:47 (1789299347) [ 1903.872548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1903.899478] alg: No test for adler32 (adler32-zlib) [ 1904.922814] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 1905.163676] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1906.872945] Key type lgssc registered [ 1908.598629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1931.999559] Key type lgssc unregistered [ 1932.333708] LNet: 98799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.346566] LNetError: 98799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1932.378733] LNet: Removed LNI 192.168.202.44@tcp [ 1933.394139] Key type .llcrypt unregistered [ 1933.400419] Key type ._llcrypt unregistered [ 1952.034639] Key type ._llcrypt registered [ 1952.037970] Key type .llcrypt registered [ 1954.030584] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 07:36:39 (1789299399) [ 1954.601628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1954.634369] alg: No test for adler32 (adler32-zlib) [ 1955.695779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1955.942155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1956.058193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1956.157349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1956.235225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1964.378246] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 07:36:49 (1789299409) [ 1964.665512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.700911] LNet: 100213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.738405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1964.809364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1964.852280] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1965.701189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.731177] LNet: 100357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1965.741398] LNetError: 100357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1965.750645] LNet: Removed LNI 192.168.202.44@tcp [ 1965.770404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.829044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1965.856189] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1966.597643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1966.639515] LNetError: 100385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.680246] LNet: Removed LNI 192.168.202.44@tcp [ 1966.742785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1966.886645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1966.962808] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1973.802188] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 07:36:59 (1789299419) [ 1973.971898] LNet: 100974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.990056] LNet: 100974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1974.012374] LNetError: 100974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1974.032659] LNet: Removed LNI 192.168.202.44@tcp [ 1974.793561] Key type .llcrypt unregistered [ 1974.797500] Key type ._llcrypt unregistered [ 1980.709102] Key type ._llcrypt registered [ 1980.712312] Key type .llcrypt registered [ 1980.876822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1980.900277] alg: No test for adler32 (adler32-zlib) [ 1981.892588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1982.024825] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 1982.028979] LNet: *** cfs_fail_loc=e003, val=100*** [ 1983.236487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 1983.260137] LNet: *** cfs_fail_loc=e003, val=100*** [ 1983.263832] LNet: Skipped 2 previous similar messages [ 1983.271528] Ping reply received with mapping: [ 1983.277245] found metadata reply NI[192.168.202.44@tcp] [ 1983.282826] 2c ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1983.284438] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1983.295574] 5a 5a 5a 5a 5a 5a 5a 5a [ 1983.599511] LNet: 101370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1983.606569] LNetError: 101370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1983.619909] LNet: Removed LNI 192.168.202.44@tcp [ 1984.371475] Key type .llcrypt unregistered [ 1984.375490] Key type ._llcrypt unregistered [ 1996.447115] Key type ._llcrypt registered [ 1996.450369] Key type .llcrypt registered [ 1997.610585] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 07:37:23 (1789299443) [ 1998.142484] Key type .llcrypt unregistered [ 1998.145538] Key type ._llcrypt unregistered [ 2005.124430] Key type ._llcrypt registered [ 2005.128746] Key type .llcrypt registered [ 2005.339331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2005.356399] alg: No test for adler32 (adler32-zlib) [ 2006.487625] LNetError: 102408:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2006.920427] Key type .llcrypt unregistered [ 2006.922109] Key type ._llcrypt unregistered [ 2011.725610] Key type ._llcrypt registered [ 2011.727472] Key type .llcrypt registered [ 2012.906404] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 07:37:38 (1789299458) [ 2013.448653] Key type .llcrypt unregistered [ 2013.450611] Key type ._llcrypt unregistered [ 2019.626737] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2019.648671] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2022.243573] Key type ._llcrypt registered [ 2022.245167] Key type .llcrypt registered [ 2022.389815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2022.399888] alg: No test for adler32 (adler32-zlib) [ 2023.330522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2023.419398] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2032.703568] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 07:37:58 (1789299478) [ 2032.809779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2032.841298] LNet: 104176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.865472] LNetError: 104176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2032.889035] LNet: Removed LNI 10.1.2.3@tcp [ 2032.912499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.550470] Key type .llcrypt unregistered [ 2033.552143] Key type ._llcrypt unregistered [ 2039.210431] Key type ._llcrypt registered [ 2039.212962] Key type .llcrypt registered [ 2039.408162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2039.427438] alg: No test for adler32 (adler32-zlib) [ 2040.360063] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2040.420124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 2040.557629] LNet: 104467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2040.564399] LNetError: 104467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.650107] LNet: Removed LNI 192.168.202.44@tcp [ 2042.320428] Key type .llcrypt unregistered [ 2042.322648] Key type ._llcrypt unregistered [ 2053.830444] Key type ._llcrypt registered [ 2053.832787] Key type .llcrypt registered [ 2055.397814] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 07:38:20 (1789299500) [ 2055.951523] Key type .llcrypt unregistered [ 2055.954682] Key type ._llcrypt unregistered [ 2061.448203] Key type ._llcrypt registered [ 2061.451401] Key type .llcrypt registered [ 2061.773127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2061.808086] alg: No test for adler32 (adler32-zlib) [ 2068.834222] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 07:38:34 (1789299514) [ 2068.946447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2068.982744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2069.054803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2069.097132] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2069.624659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2069.650543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2069.678617] LNet: Removed LNI 192.168.202.44@tcp [ 2069.876656] LNet: 106206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2070.222565] Key type .llcrypt unregistered [ 2070.226124] Key type ._llcrypt unregistered [ 2080.833404] Key type ._llcrypt registered [ 2080.834504] Key type .llcrypt registered [ 2081.837177] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 07:38:48 (1789299528) [ 2082.104451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2082.114265] alg: No test for adler32 (adler32-zlib) [ 2083.036966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2083.102325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2083.133208] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2083.866382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 2083.898836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.44@tcp [ 2083.924258] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2083.927246] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2083.930303] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2083, 0, 0 [ 2083.933957] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2083.937456] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2083.942008] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2083.945622] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2083, 0, 0 [ 2083.950828] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2083.954852] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2083.959568] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2083.963071] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2083, 0, 2133 [ 2083.968266] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2083.971381] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2083.975560] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2083.979297] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2083, 0, 2133 [ 2083.984387] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2090.652903] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 07:38:56 (1789299536) [ 2091.498970] LNet: 107742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.508388] LNetError: 107742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2092.582882] LNet: Removed LNI 192.168.202.44@tcp [ 2093.207707] Key type .llcrypt unregistered [ 2093.213611] Key type ._llcrypt unregistered [ 2099.197523] Key type ._llcrypt registered [ 2099.199808] Key type .llcrypt registered [ 2099.385994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2099.426915] alg: No test for adler32 (adler32-zlib) [ 2100.347314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2100.389993] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2109.365350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.144@tcp [ 2120.630548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2120.674392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2120.691237] LNet: Added LNI 192.168.202.44@tcp1 [8/256/0/180] [ 2121.700444] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2121.932980] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2121.937884] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2121.944610] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2124.421740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2124.449501] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2135.566380] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 07:39:41 (1789299581) [ 2135.704167] LNet: 109240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.713811] LNetError: 109240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2135.737212] LNet: Removed LNI 192.168.202.44@tcp [ 2136.799219] LNet: Removed LNI 192.168.202.44@tcp1 [ 2136.806771] LNetError: 109240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2136.815338] LNetError: 109240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2136.834909] LNet: Removed LNI 10.1.2.3@tcp99 [ 2137.566748] Key type .llcrypt unregistered [ 2137.569844] Key type ._llcrypt unregistered [ 2144.555266] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2144.558303] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2147.093674] Key type ._llcrypt registered [ 2147.095188] Key type .llcrypt registered [ 2147.356321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2147.370869] alg: No test for adler32 (adler32-zlib) [ 2148.371784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.490452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2148.507767] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2149.315392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2149.350053] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2151.934361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2152.004920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2152.075333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2152.115335] LNetError: 109990:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2152.167973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2152.224858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2152.713899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2152.819788] LNet: 110010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.836465] LNetError: 110010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2152.850945] LNet: Removed LNI 10.1.2.3@tcp [ 2153.900765] LNet: Removed LNI 192.168.202.44@tcp [ 2154.706624] Key type .llcrypt unregistered [ 2154.711905] Key type ._llcrypt unregistered [ 2170.569936] Key type ._llcrypt registered [ 2170.572093] Key type .llcrypt registered [ 2172.259612] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 07:40:17 (1789299617) [ 2172.774653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2172.794202] alg: No test for adler32 (adler32-zlib) [ 2173.822440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2173.901621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2173.931525] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2174.666764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 2174.749685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg244-client@tcp [ 2183.571286] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 07:40:28 (1789299628) [ 2183.681773] LNet: 111532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2183.691736] LNetError: 111532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2183.711089] LNet: Removed LNI 192.168.202.44@tcp [ 2184.376235] Key type .llcrypt unregistered [ 2184.380914] Key type ._llcrypt unregistered [ 2192.049805] Key type ._llcrypt registered [ 2192.052541] Key type .llcrypt registered [ 2192.226691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2192.238200] alg: No test for adler32 (adler32-zlib) [ 2193.144508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2193.171349] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2193.210562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2199.260672] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 07:40:45 (1789299645) [ 2200.525673] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2202.217313] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 07:40:48 (1789299648) [ 2203.392406] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2205.179419] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 07:40:50 (1789299650) [ 2205.259870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2205.300149] LNet: 112954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2205.309061] LNetError: 112954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2205.329551] LNet: Removed LNI 192.168.202.44@tcp [ 2205.356266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.411574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2205.422510] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2206.085798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2206.129285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2212.826640] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 07:40:58 (1789299658) [ 2212.925869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2212.951659] LNet: 113661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2212.955507] LNetError: 113661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2212.968903] LNet: Removed LNI 192.168.202.44@tcp [ 2212.994988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2213.048262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2213.089751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2213.605435] Key type .llcrypt unregistered [ 2213.606808] Key type ._llcrypt unregistered [ 2225.205601] Key type ._llcrypt registered [ 2225.207776] Key type .llcrypt registered [ 2226.501377] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 07:41:12 (1789299672) [ 2226.845175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2226.858905] alg: No test for adler32 (adler32-zlib) [ 2227.733339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.763096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2227.780704] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2228.341591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2228.407317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2233.932713] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 07:41:20 (1789299680) [ 2234.076673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.112126] LNet: 115217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.120204] LNetError: 115217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2234.137198] LNet: Removed LNI 192.168.202.44@tcp [ 2234.171342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2234.270729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2234.318508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2241.598360] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 07:41:27 (1789299687) [ 2241.682954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2241.715273] LNet: 115795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2241.742879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.855507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2242.210879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2242.262735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2242.356483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2242.851699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2242.888735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2242.992920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2243.032394] LNet: 115836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2243.072633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.142880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2243.656291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2243.690482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2243.803577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2244.161794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2244.198227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2250.414363] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 07:41:36 (1789299696) [ 2250.722108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2250.761520] LNet: 116436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2250.817189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2250.946277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2250.998803] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2252.093636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2252.127847] LNet: Added LNI 192.168.202.44@tcp2 [8/256/0/180] [ 2253.621488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2253.694883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2254.014988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2254.040843] LNetError: 116716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2254.050779] LNet: Removed LNI 192.168.202.44@tcp [ 2254.064554] LNet: Removed LNI 192.168.202.44@tcp2 [ 2254.088688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.144709] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2254.153063] LNet: Added LNI 192.168.202.44@tcp2 [8/256/0/180] [ 2254.157829] LNet: 116722:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2254.771566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2255.057685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2255.093329] LNet: 116759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2255.106164] LNet: 116759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2255.114754] LNetError: 116759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2255.121823] LNetError: 116759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2255.128824] LNet: Removed LNI 192.168.202.44@tcp [ 2255.146599] LNet: Removed LNI 192.168.202.44@tcp2 [ 2255.212954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2255.289045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2255.324786] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2256.173333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2256.205423] LNet: Added LNI 192.168.202.44@tcp2 [8/256/0/180] [ 2257.180302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2257.231745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2257.309499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2257.589409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2257.619501] LNetError: 117044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2257.626021] LNetError: 117044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2257.631592] LNet: Removed LNI 192.168.202.44@tcp [ 2258.666965] LNet: Removed LNI 192.168.202.44@tcp2 [ 2258.709127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2258.832318] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2258.838607] LNet: Added LNI 192.168.202.44@tcp2 [8/256/0/180] [ 2258.856744] LNet: 117050:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2259.745369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2260.036955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2260.071483] LNet: Removed LNI 192.168.202.44@tcp [ 2261.095839] LNet: Removed LNI 192.168.202.44@tcp2 [ 2261.129188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2261.250091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2261.298160] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2262.406673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2262.428324] LNet: Added LNI 192.168.202.44@tcp2 [8/256/0/180] [ 2263.059958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2263.113223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2263.347603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2263.372909] LNet: 117366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.379584] LNet: 117366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2263.387967] LNetError: 117366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2263.394723] LNetError: 117366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2263.406890] LNet: Removed LNI 192.168.202.44@tcp [ 2263.429158] LNet: Removed LNI 192.168.202.44@tcp2 [ 2263.450885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2263.507372] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2263.520617] LNet: Added LNI 192.168.202.44@tcp2 [8/256/0/180] [ 2263.526632] LNet: 117372:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2264.229856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2264.379838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2265.439883] LNet: Removed LNI 192.168.202.44@tcp [ 2265.454866] LNet: Removed LNI 192.168.202.44@tcp2 [ 2265.485962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2265.575585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2265.601080] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2266.339963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2266.385682] LNet: Added LNI 192.168.202.44@tcp2 [8/256/0/180] [ 2267.533365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2267.587686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2267.649978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2267.910142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2268.960673] LNet: Removed LNI 192.168.202.44@tcp [ 2269.988977] LNet: Removed LNI 192.168.202.44@tcp2 [ 2270.008461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2270.051685] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2270.055855] LNet: Added LNI 192.168.202.44@tcp2 [8/256/0/180] [ 2270.059569] LNet: 117700:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2270.388848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2270.547168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.601231] LNet: Removed LNI 192.168.202.44@tcp [ 2271.661928] LNet: Removed LNI 192.168.202.44@tcp2 [ 2271.706502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.802942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2271.834757] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2272.746390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2272.761846] LNet: Added LNI 192.168.202.44@tcp2 [8/256/0/180] [ 2273.933803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2273.961643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2274.032552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2274.330369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2274.363932] LNetError: 118034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2274.368844] LNetError: 118034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2274.375284] LNet: Removed LNI 192.168.202.44@tcp [ 2274.387508] LNet: Removed LNI 192.168.202.44@tcp2 [ 2274.404761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2274.453884] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2274.459314] LNet: Added LNI 192.168.202.44@tcp2 [8/256/0/180] [ 2274.462851] LNet: 118040:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2275.178337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2275.624784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2276.703428] LNet: Removed LNI 192.168.202.44@tcp [ 2276.723203] LNet: Removed LNI 192.168.202.44@tcp2 [ 2276.773557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2276.879245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2276.908532] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2277.841436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2277.864888] LNet: Added LNI 192.168.202.44@tcp2 [8/256/0/180] [ 2279.290121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2279.327150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2279.382698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2279.448778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2280.001631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2280.048930] LNet: 118394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2280.055246] LNet: 118394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2281.119227] LNet: Removed LNI 192.168.202.44@tcp [ 2282.153550] LNet: Removed LNI 192.168.202.44@tcp2 [ 2282.245635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.343465] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2282.350191] LNet: Added LNI 192.168.202.44@tcp2 [8/256/0/180] [ 2283.167554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2291.076102] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2293.128511] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 07:42:18 (1789299738) [ 2293.289672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2293.342082] LNetError: 119154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2293.349925] LNetError: 119154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2293.363041] LNet: Removed LNI 192.168.202.44@tcp [ 2294.438313] LNet: Removed LNI 192.168.202.44@tcp2 [ 2294.458760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2294.750781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2294.788809] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2295.715803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2295.730044] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2296.631515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2296.663790] LNet: Removed LNI 10.1.2.3@tcp [ 2296.674544] LNet: Removed LNI 192.168.202.44@tcp [ 2296.721993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2296.832838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2296.868728] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2296.878585] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2297.371726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2297.392690] LNet: Removed LNI 192.168.202.44@tcp [ 2297.403732] LNet: Removed LNI 10.1.2.3@tcp [ 2303.868268] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 07:42:30 (1789299750) [ 2303.951770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2303.998799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.107953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2304.133797] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2304.702611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2304.747317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 2304.783045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 2304.817700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 2304.847315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 2304.873897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 2304.900336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 2304.928079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 2304.955650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 2305.015482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 2305.073443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.44@tcp [ 2305.213434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2310.647534] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 07:42:36 (1789299756) [ 2310.792505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2311.846790] LNet: Removed LNI 192.168.202.44@tcp [ 2311.876509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.006980] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2312.095919] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2312.109097] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2312.118117] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2314.664942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2314.688103] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2315.466704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2315.493555] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2317.273864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2317.334941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2317.437927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.44@tcp -m PUT -e local_timeout -r 1 [ 2317.543494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2317.602628] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2317.608645] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2318.018471] LNetError: 120840:0:(lib-move.c:4801:lnet_parse()) 192.168.202.44@tcp, src 192.168.202.44@tcp: Dropping PUT (error -108 looking up sender) [ 2319.071686] LNet: 121298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2319.092680] LNet: 121298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2319.121880] LNet: Removed LNI 10.1.2.3@tcp [ 2320.166738] LNet: Removed LNI 192.168.202.44@tcp [ 2320.635571] Key type .llcrypt unregistered [ 2320.638313] Key type ._llcrypt unregistered [ 2333.668900] Key type ._llcrypt registered [ 2333.671516] Key type .llcrypt registered [ 2335.316626] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 07:43:00 (1789299780) [ 2335.787198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2335.801768] alg: No test for adler32 (adler32-zlib) [ 2336.768344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.246050] LNet: 122147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.984444] Key type .llcrypt unregistered [ 2337.987352] Key type ._llcrypt unregistered [ 2344.820573] Key type ._llcrypt registered [ 2344.826168] Key type .llcrypt registered [ 2345.030618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2345.051761] alg: No test for adler32 (adler32-zlib) [ 2345.941851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2345.959216] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2352.869878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.144@tcp [ 2353.075396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2353.630154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.144@tcp [ 2353.894082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2353.940346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2353.966243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2353.990211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2354.017844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2354.078278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2354.110777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2354.144238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2354.159349] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.144@tcp [ 2354.201296] LNet: 122594:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2354.243291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2354.302581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2354.395485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2354.476910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2354.538746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2354.615217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2355.473398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2355.736353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2355.774630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2355.829307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2355.879592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2355.919680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2355.947735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2355.984644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2356.018751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2356.042406] LNet: 122711:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2356.088514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2356.143638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2356.177427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2356.196193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2356.236409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2356.263517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2356.966456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.144@tcp [ 2357.332136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2357.364876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2357.392383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2357.423497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2357.460334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2357.505309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2357.535704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2357.589831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2357.629915] LNet: 122828:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2357.653949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2357.695950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2357.774266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2357.842643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2357.914264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2357.971394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2358.643300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2358.887796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2358.922915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2358.959951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2358.983634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2359.003764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2359.032324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2359.054915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2359.081568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2359.109522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2359.132675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2359.162284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2359.205670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2359.240470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2359.268507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2359.529140] LNet: 122992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2359.536683] LNetError: 122992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2359.552592] LNet: Removed LNI 192.168.202.44@tcp [ 2360.017313] Key type .llcrypt unregistered [ 2360.021726] Key type ._llcrypt unregistered [ 2376.430624] Key type ._llcrypt registered [ 2376.432834] Key type .llcrypt registered [ 2378.208862] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 07:43:43 (1789299823) [ 2378.616335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2378.627946] alg: No test for adler32 (adler32-zlib) [ 2379.510175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2379.849271] LNet: 123876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2380.625404] Key type .llcrypt unregistered [ 2380.632711] Key type ._llcrypt unregistered [ 2388.605211] Key type ._llcrypt registered [ 2388.609720] Key type .llcrypt registered [ 2388.991881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2389.030552] alg: No test for adler32 (adler32-zlib) [ 2390.088812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2390.137961] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2400.297380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.144@tcp [ 2411.690164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2411.753670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2411.789055] LNet: Added LNI 192.168.202.44@tcp1 [8/256/0/180] [ 2412.968312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2413.517528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.144@tcp [ 2413.732108] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.144@tcp [ 2413.740081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2413.769101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2413.799238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2413.827352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2413.854124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2413.881080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2413.910153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2413.936637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2413.952897] LNet: 124526:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2413.968941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2413.994572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2414.026096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2414.055209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2414.085947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2414.112924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2414.730542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2414.992523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2415.023603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2415.051939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2415.080338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2415.113268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2415.139284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2415.166502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2415.194898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2415.210918] LNet: 124643:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2415.228960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2415.258966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2415.291329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2415.320455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2415.359317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2415.391211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2415.957367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.144@tcp [ 2416.149846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2416.173433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2416.195289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2416.224989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2416.254319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2416.278333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2416.302242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2416.328636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2416.339939] LNet: 124760:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2416.356187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2416.395495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2416.444252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2416.489104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2416.548976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2416.617225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2417.414782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2417.775940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2417.845124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2417.901159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2417.969186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2418.040935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2418.105950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2418.181370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2418.249455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2418.299248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2418.360577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2418.412746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2418.454200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2418.508444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2418.561529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2424.618836] LNet: 124958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.622375] LNetError: 124958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2424.627325] LNet: Removed LNI 192.168.202.44@tcp [ 2424.634923] LNet: Removed LNI 192.168.202.44@tcp1 [ 2425.157858] Key type .llcrypt unregistered [ 2425.163748] Key type ._llcrypt unregistered [ 2445.752370] Key type ._llcrypt registered [ 2445.754432] Key type .llcrypt registered [ 2447.211202] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 07:44:52 (1789299892) [ 2448.015542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2448.034437] alg: No test for adler32 (adler32-zlib) [ 2449.028570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2449.086574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2449.135613] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2450.073703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2450.096119] LNet: Added LNI 192.168.202.44@tcp2 [8/256/0/180] [ 2450.895575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.44@tcp [ 2450.976441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2452.063533] LNet: Removed LNI 192.168.202.44@tcp [ 2458.322480] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 07:45:04 (1789299904) [ 2459.571623] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2461.620423] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 07:45:07 (1789299907) [ 2469.489144] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 07:45:15 (1789299915) [ 2476.472724] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 07:45:22 (1789299922) [ 2482.228018] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 07:45:28 (1789299928) [ 2489.170704] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 07:45:35 (1789299935) [ 2496.913745] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 07:45:42 (1789299942) [ 2504.063205] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 07:45:49 (1789299949) [ 2510.409181] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 07:45:56 (1789299956) [ 2517.350583] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 07:46:03 (1789299963) [ 2525.136220] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 07:46:11 (1789299971) [ 2532.129619] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 07:46:17 (1789299977) [ 2539.564628] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 07:46:25 (1789299985) [ 2545.629591] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 07:46:31 (1789299991) [ 2552.151252] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 07:46:37 (1789299997) [ 2560.309511] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 07:46:46 (1789300006) [ 2567.764669] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 07:46:53 (1789300013) [ 2574.662269] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 07:47:00 (1789300020) [ 2582.229345] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 07:47:08 (1789300028) [ 2582.388655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2582.448393] LNet: 136647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2582.455738] LNetError: 136647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2583.539259] LNet: Removed LNI 192.168.202.44@tcp2 [ 2583.575995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2583.787822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2583.824179] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2585.110994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2585.151669] LNet: 136789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2585.168459] LNetError: 136789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2585.190355] LNet: Removed LNI 192.168.202.44@tcp [ 2585.251406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2585.351995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2585.392977] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2592.611936] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 07:47:18 (1789300038) [ 2600.505763] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 07:47:25 (1789300045) [ 2607.740799] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 07:47:33 (1789300053) [ 2614.883468] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 07:47:40 (1789300060) [ 2621.400592] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 07:47:47 (1789300067) [ 2628.808883] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 07:47:54 (1789300074) [ 2635.874478] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 07:48:01 (1789300081) [ 2643.092249] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 07:48:09 (1789300089) [ 2651.190966] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 07:48:16 (1789300096) [ 2660.032570] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 07:48:25 (1789300105) [ 2666.526773] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 07:48:32 (1789300112) [ 2673.073640] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 07:48:39 (1789300119) [ 2681.011664] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2389 sec ========== 07:48:46 (1789300126) [ 2681.348797] LNet: 144255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2681.356383] LNetError: 144255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2681.375707] LNet: Removed LNI 192.168.202.44@tcp [ 2682.163733] Key type .llcrypt unregistered [ 2682.174103] Key type ._llcrypt unregistered [ 2694.868100] Key type ._llcrypt registered [ 2694.876075] Key type .llcrypt registered [ 2695.334986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2695.343812] alg: No test for adler32 (adler32-zlib) [ 2696.539526] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 2696.836370] LNet: Added LNI 192.168.202.44@tcp [8/256/0/180] [ 2698.516551] Key type lgssc registered [ 2699.658192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2755.386929] Lustre: Mounted lustre-client - version 2.17.57_108_gfed9771 [ 2760.906264] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2768.708179] Lustre: Unmounted lustre-client [ 2805.181176] Key type lgssc unregistered [ 2805.420044] LNet: 147202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2805.435459] LNetError: 147202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2805.460538] LNet: Removed LNI 192.168.202.44@tcp [ 2806.135162] Key type .llcrypt unregistered [ 2806.137303] Key type ._llcrypt unregistered