[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 482815980 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002318] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007524] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010121] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.012031] SELinux: Initializing. [ 0.013073] *** VALIDATE selinux *** [ 0.020735] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025323] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028027] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030107] *** VALIDATE tmpfs *** [ 0.032038] *** VALIDATE proc *** [ 0.033152] *** VALIDATE cgroup *** [ 0.034007] *** VALIDATE cgroup2 *** [ 0.036141] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037140] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039024] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.042714] debug: unmapping init [mem 0xffffffff9c059000-0xffffffff9c060fff] [ 0.044165] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045672] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046022] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048009] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050013] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053275] rcu: Hierarchical SRCU implementation. [ 0.055318] smp: Bringing up secondary CPUs ... [ 0.056458] x86: Booting SMP configuration: [ 0.057013] .... node #0, CPUs: #1 #2 #3 [ 0.060007] smp: Brought up 1 node, 4 CPUs [ 0.062009] smpboot: Max logical packages: 1 [ 0.063016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.101480] node 0 deferred pages initialised in 36ms [ 0.105152] devtmpfs: initialized [ 0.106242] x86/mm: Memory block size: 128MB [ 0.109721] gcov: version magic: 0x41383552 [ 0.110313] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.113150] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.114313] pinctrl core: initialized pinctrl subsystem [ 0.116192] [ 0.116662] ************************************************************* [ 0.118010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.120008] ** ** [ 0.122012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.123007] ** ** [ 0.125010] ** This means that this kernel is built to expose internal ** [ 0.127008] ** IOMMU data structures, which may compromise security on ** [ 0.128007] ** your system. ** [ 0.129009] ** ** [ 0.131009] ** If you see this message and you are not debugging the ** [ 0.133009] ** kernel, report this immediately to your vendor! ** [ 0.134011] ** ** [ 0.136015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.138008] ************************************************************* [ 0.139579] NET: Registered protocol family 16 [ 0.142558] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.147064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.151062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.156104] cpuidle: using governor menu [ 0.157657] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.160566] PCI: Using configuration type 1 for base access [ 0.162131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.172100] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.175022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.181076] cryptd: max_cpu_qlen set to 1000 [ 0.185237] ACPI: Added _OSI(Module Device) [ 0.186020] ACPI: Added _OSI(Processor Device) [ 0.188012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.190009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.194873] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.201410] ACPI: Interpreter enabled [ 0.203054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.204011] ACPI: Using IOAPIC for interrupt routing [ 0.206082] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.209398] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.220387] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.222031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.225014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.233474] acpiphp: Slot [2] registered [ 0.234145] acpiphp: Slot [5] registered [ 0.236096] acpiphp: Slot [6] registered [ 0.237100] acpiphp: Slot [3] registered [ 0.239079] acpiphp: Slot [4] registered [ 0.241079] acpiphp: Slot [7] registered [ 0.242092] acpiphp: Slot [8] registered [ 0.244089] acpiphp: Slot [9] registered [ 0.246073] acpiphp: Slot [10] registered [ 0.247087] acpiphp: Slot [11] registered [ 0.249093] acpiphp: Slot [12] registered [ 0.250071] acpiphp: Slot [13] registered [ 0.252077] acpiphp: Slot [14] registered [ 0.253137] acpiphp: Slot [15] registered [ 0.255074] acpiphp: Slot [16] registered [ 0.256072] acpiphp: Slot [17] registered [ 0.258084] acpiphp: Slot [18] registered [ 0.260085] acpiphp: Slot [19] registered [ 0.261093] acpiphp: Slot [20] registered [ 0.263086] acpiphp: Slot [21] registered [ 0.265098] acpiphp: Slot [22] registered [ 0.267096] acpiphp: Slot [23] registered [ 0.268116] acpiphp: Slot [24] registered [ 0.270097] acpiphp: Slot [25] registered [ 0.272089] acpiphp: Slot [26] registered [ 0.273157] acpiphp: Slot [27] registered [ 0.275104] acpiphp: Slot [28] registered [ 0.276079] acpiphp: Slot [29] registered [ 0.278143] acpiphp: Slot [30] registered [ 0.279105] acpiphp: Slot [31] registered [ 0.280103] PCI host bridge to bus 0000:00 [ 0.282013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.284017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.286012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.288015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.290015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.293010] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.295214] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.297850] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.302079] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.310013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.315045] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.318017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.321014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.324020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.327597] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.330964] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.333042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.336850] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.341011] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.351710] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.356012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.361773] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.370016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.382015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.399014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.408311] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.419016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.428025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.454017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.475760] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.478324] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.481262] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.484329] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.487191] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.491109] iommu: Default domain type: Passthrough [ 0.493292] SCSI subsystem initialized [ 0.494111] ACPI: bus type USB registered [ 0.495085] usbcore: registered new interface driver usbfs [ 0.497059] usbcore: registered new interface driver hub [ 0.499060] usbcore: registered new device driver usb [ 0.500124] pps_core: LinuxPPS API ver. 1 registered [ 0.502008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.504070] PTP clock support registered [ 0.506136] EDAC MC: Ver: 3.0.0 [ 0.509161] PCI: Using ACPI for IRQ routing [ 0.510429] NetLabel: Initializing [ 0.511007] NetLabel: domain hash size = 128 [ 0.512010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.514065] NetLabel: unlabeled traffic allowed by default [ 0.516127] vgaarb: loaded [ 0.517250] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.519012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.523346] clocksource: Switched to clocksource kvm-clock [ 0.636652] VFS: Disk quotas dquot_6.6.0 [ 0.638663] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.641905] *** VALIDATE ramfs *** [ 0.643592] *** VALIDATE hugetlbfs *** [ 0.645528] pnp: PnP ACPI init [ 0.648125] pnp: PnP ACPI: found 6 devices [ 0.664903] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.668362] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.670568] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.672749] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.675230] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.677639] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.679717] NET: Registered protocol family 2 [ 0.681666] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.685315] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.687848] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.692073] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.694900] TCP: Hash tables configured (established 65536 bind 65536) [ 0.697545] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.701479] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.704672] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.708641] NET: Registered protocol family 1 [ 0.711358] RPC: Registered named UNIX socket transport module. [ 0.713414] RPC: Registered udp transport module. [ 0.715205] RPC: Registered tcp transport module. [ 0.717041] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.719334] NET: Registered protocol family 44 [ 0.720927] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.723323] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.725324] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.727551] PCI: CLS 0 bytes, default 64 [ 0.729638] Unpacking initramfs... [ 2.124900] debug: unmapping init [mem 0xffff909d7cc64000-0xffff909d7ffcffff] [ 2.129201] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.131152] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.134069] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.664807] Initialise system trusted keyrings [ 2.666602] Key type blacklist registered [ 2.668550] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.678141] zbud: loaded [ 2.681402] *** VALIDATE nfs *** [ 2.682824] *** VALIDATE nfs4 *** [ 2.684746] pstore: using deflate compression [ 2.694913] Platform Keyring initialized [ 2.832240] NET: Registered protocol family 38 [ 2.833840] Key type asymmetric registered [ 2.835529] Asymmetric key parser 'x509' registered [ 2.836859] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.839763] io scheduler mq-deadline registered [ 2.841951] io scheduler kyber registered [ 2.843992] io scheduler bfq registered [ 2.847638] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.850023] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.852644] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.855381] ACPI: Power Button [PWRF] [ 2.861094] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.867169] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.905540] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.934357] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.965069] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.969591] Non-volatile memory driver v1.3 [ 2.971591] Linux agpgart interface v0.103 [ 3.010638] virtio_blk virtio1: [vda] 145880 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.013730] vda: detected capacity change from 0 to 74690560 [ 3.061397] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.064519] vdb: detected capacity change from 0 to 1073741824 [ 3.100139] libphy: Fixed MDIO Bus: probed [ 3.113563] usbcore: registered new interface driver usbserial_generic [ 3.115808] usbserial: USB Serial support registered for generic [ 3.118811] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.124159] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.126229] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.128749] mousedev: PS/2 mouse device common for all mice [ 3.131862] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.135886] rtc_cmos 00:05: RTC can wake from S4 [ 3.140100] rtc_cmos 00:05: registered as rtc0 [ 3.140245] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.141786] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.148566] intel_pstate: CPU model not supported [ 3.148887] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.154590] hid: raw HID events driver (C) Jiri Kosina [ 3.158362] usbcore: registered new interface driver usbhid [ 3.160908] usbhid: USB HID core driver [ 3.162289] drop_monitor: Initializing network drop monitor service [ 3.165588] Initializing XFRM netlink socket [ 3.167541] NET: Registered protocol family 10 [ 3.171093] Segment Routing with IPv6 [ 3.172520] NET: Registered protocol family 17 [ 3.175653] mpls_gso: MPLS GSO support [ 3.184927] RAS: Correctable Errors collector initialized. [ 3.187236] AVX version of gcm_enc/dec engaged. [ 3.189033] AES CTR mode by8 optimization enabled [ 3.289094] sched_clock: Marking stable (3289073058, 0)->(4156363337, -867290279) [ 3.293178] registered taskstats version 1 [ 3.296752] Loading compiled-in X.509 certificates [ 3.299214] zswap: loaded using pool lzo/zbud [ 3.331643] Key type big_key registered [ 3.347886] Key type encrypted registered [ 3.349487] ima: No TPM chip found, activating TPM-bypass! [ 3.351533] ima: Allocated hash algorithm: sha1 [ 3.353271] ima: No architecture policies found [ 3.354901] evm: Initialising EVM extended attributes: [ 3.356625] evm: security.selinux [ 3.357852] evm: security.ima [ 3.358848] evm: security.capability [ 3.360144] evm: HMAC attrs: 0x1 [ 3.363758] rtc_cmos 00:05: setting system clock to 2026-08-03 16:15:51 UTC (1785773751) [ 3.369117] debug: unmapping init [mem 0xffffffff9d003000-0xffffffff9d1fffff] [ 3.372492] debug: unmapping init [mem 0xffffffff9bd82000-0xffffffff9c058fff] [ 3.382134] Write protecting the kernel read-only data: 28672k [ 3.386026] debug: unmapping init [mem 0xffffffff9a403000-0xffffffff9a5fffff] [ 3.388701] debug: unmapping init [mem 0xffffffff9ad14000-0xffffffff9adfffff] [ 3.423132] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.431670] systemd[1]: Detected virtualization kvm. [ 3.433539] systemd[1]: Detected architecture x86-64. [ 3.435428] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.463488] systemd[1]: No hostname configured. [ 3.465265] systemd[1]: Set hostname to . [ 3.467342] random: systemd: uninitialized urandom read (16 bytes read) [ 3.469582] systemd[1]: Initializing machine ID from random generator. [ 3.517181] random: ln: uninitialized urandom read (6 bytes read) [ 3.603790] random: systemd: uninitialized urandom read (16 bytes read) [ 3.606482] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.611304] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.615587] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.178824] device-mapper: uevent: version 1.0.3 [ 4.181894] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.875880] virtio_net virtio0 ens2: renamed from eth0 [ 4.939943] scsi host0: ata_piix [ 4.959614] scsi host1: ata_piix [ 4.985170] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.988808] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.221258] dracut-initqueue[597]: RTNETLINK answers: File exists [ 9.865798] random: crng init done [ 9.869675] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.136185] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.177113] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.480916] SELinux: Disabled at runtime. [ 11.555651] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.568745] systemd[1]: Detected virtualization kvm. [ 11.570819] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.095210] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.099373] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.104822] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.111318] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.114766] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.121022] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.125139] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ 12.200807] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.539240] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.816549] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.856286] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.934103] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.943528] EDAC sbridge: Ver: 1.1.2 [ 13.897800] Key type dns_resolver registered [ 14.205365] NFS: Registering the id_resolver key type [ 14.206758] Key type id_resolver registered [ 14.207982] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg118-client login: [ 40.151118] libcfs: loading out-of-tree module taints kernel. [ 40.196198] Key type ._llcrypt registered [ 40.197955] Key type .llcrypt registered [ 40.522700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 40.528610] alg: No test for adler32 (adler32-zlib) [ 41.543285] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 41.883789] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 43.520146] Key type lgssc registered [ 44.234746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 203.068884] Lustre: Mounted lustre-client [ 209.173121] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 224.366592] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing check_logdir /tmp/testlogs/ [ 228.834290] Lustre: lustre-OST0000-osc-ffff909dd8330000: disconnect after 22s idle [ 231.218955] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing yml_node [ 237.563451] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 241.335666] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 244.413922] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 246.661806] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 3 12:19:52 EDT 2026 [ 266.744367] Lustre: Unmounted lustre-client [ 299.791285] Key type lgssc unregistered [ 300.128839] LNet: 8390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.132882] LNetError: 8390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 300.149696] LNet: Removed LNI 192.168.201.18@tcp [ 301.248344] Key type .llcrypt unregistered [ 301.249731] Key type ._llcrypt unregistered [ 348.357519] Key type ._llcrypt registered [ 348.368303] Key type .llcrypt registered [ 349.026192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 349.102623] alg: No test for adler32 (adler32-zlib) [ 350.072521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 350.143372] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 350.174785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 357.060603] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 357.240448] LNet: 9270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.250824] LNetError: 9270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 357.327523] LNet: Removed LNI 192.168.201.18@tcp [ 358.458602] Key type .llcrypt unregistered [ 358.467933] Key type ._llcrypt unregistered [ 368.641920] Key type ._llcrypt registered [ 368.643447] Key type .llcrypt registered [ 370.833092] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:21:56 (1785774116) [ 371.486982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 371.509863] alg: No test for adler32 (adler32-zlib) [ 372.529877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.642931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.680364] LNet: 9693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.702594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.817826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 372.872062] LNet: 9700:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 382.393935] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:22:08 (1785774128) [ 382.707139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.747468] LNet: 10279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.781482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.928752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 392.992933] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:22:19 (1785774139) [ 393.316198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.358250] LNet: 10864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.397875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.547940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 402.226617] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:22:29 (1785774149) [ 402.607378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.645968] LNet: 11450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.668555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.795831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 412.078782] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:22:38 (1785774158) [ 412.340392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.385944] LNet: 12035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.408430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.534146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 413.206708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 422.497340] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:22:49 (1785774169) [ 422.625938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.678770] LNet: 12620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.726375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.895770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 423.741434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 424.198585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 430.267262] hrtimer: interrupt took 13253844 ns [ 436.212744] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:23:00 (1785774180) [ 436.491427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.666351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.964021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 452.204021] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:23:18 (1785774198) [ 452.356301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.385310] LNet: 13804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 452.394255] LNet: 13804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 452.458212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.613100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 452.672489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 453.327137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 453.545243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 453.811397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 453.834504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 454.040790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 454.062842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 454.327835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 454.354499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 454.714854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 454.740902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 455.014641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 455.031686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 455.350485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 455.416481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 456.399952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 456.462454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 457.301135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 457.337383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 457.552897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 457.577186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 457.808527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 457.837285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 458.153818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 458.197468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 464.335686] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:23:30 (1785774210) [ 464.449396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.598563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.706327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 464.765478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 476.243491] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:23:41 (1785774221) [ 476.467046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.584128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.855878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 476.921842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 485.947588] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:23:51 (1785774231) [ 486.032749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.048609] LNet: 15658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 486.052020] LNet: 15658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 486.089802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.149422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 486.173976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 494.343718] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:23:59 (1785774239) [ 494.535432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.688162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.833632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 494.942748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 505.234929] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:24:11 (1785774251) [ 505.416936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.508627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.637087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 505.673033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 513.580392] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:24:20 (1785774260) [ 514.228341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.256584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.355843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 514.407192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 523.289614] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:24:29 (1785774269) [ 523.638715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.694884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.859200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 524.658322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 536.718790] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:24:41 (1785774281) [ 536.976823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.065989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.445410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 537.954380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 547.677481] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:24:54 (1785774294) [ 547.929732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.997969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.196216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 548.541933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 558.298622] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:25:04 (1785774304) [ 558.678453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.754080] LNet: 19845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 558.772353] LNet: 19845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 558.822575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.106541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 560.593552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 572.674812] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:25:18 (1785774318) [ 572.889992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.084404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.291425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 573.407350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 584.033199] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:25:30 (1785774330) [ 584.140877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.182272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.280793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 584.377478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 595.199933] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:25:41 (1785774341) [ 595.470405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.533883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.640956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 601.905475] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:25:48 (1785774348) [ 602.053939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.131827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.238732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 610.915059] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:25:57 (1785774357) [ 611.057277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.134674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.228073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 611.543482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 619.327902] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:26:06 (1785774366) [ 619.428433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.497745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.617945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 619.661665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 628.391797] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:26:14 (1785774374) [ 628.544374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 628.643495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.770904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 628.800671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 638.062623] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:26:24 (1785774384) [ 638.583236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.659500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.812645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 638.873904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 650.330290] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:26:36 (1785774396) [ 650.583975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.704455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.949534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 651.054467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 661.396661] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:26:47 (1785774407) [ 661.528455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.624024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.764555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 661.825267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 661.853963] LNetError: 25824:0:(peer.c:2277:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 661.949492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 662.026781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 669.112887] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:26:55 (1785774415) [ 669.481604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.546556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.706527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.FvAz3uhUmZ [ 669.833434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.FvAz3uhUmZ [ 669.950469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.FvAz3uhUmZ [ 677.546119] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:27:04 (1785774424) [ 677.756545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.865731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.928367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 677.964218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 678.030604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 685.235545] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 12:27:11 (1785774431) [ 685.312601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.402572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.540878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 685.596443] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 687.675394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 690.201762] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 692.788250] Key type lgssc registered [ 695.907279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 701.808744] LNetError: 27585:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.18@tcp99 has no available nets [ 705.999608] Key type lgssc unregistered [ 706.769194] LNet: 28697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.771803] LNet: 28697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 706.794793] LNetError: 28697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.815479] LNet: Removed LNI 192.168.201.18@tcp [ 709.056461] Key type .llcrypt unregistered [ 709.066830] Key type ._llcrypt unregistered [ 719.321735] Key type ._llcrypt registered [ 719.323692] Key type .llcrypt registered [ 721.056623] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:27:47 (1785774467) [ 721.619955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 721.640044] alg: No test for adler32 (adler32-zlib) [ 722.540801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.653542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 725.670982] LNet: Message forwarding will be enabled in 2s [ 728.736127] LNet: Message forwarding enabled [ 729.406527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 729.425125] LNet: Message forwarding will stop in 2s to 6s [ 733.150945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 734.522397] LNet: Message forwarding disabled [ 737.802616] LNet: Message forwarding will be enabled in 2s [ 741.728116] LNet: Message forwarding enabled [ 749.348721] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:28:15 (1785774495) [ 749.659620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.709643] LNet: 30100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.342554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.539372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 759.780303] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:28:26 (1785774506) [ 759.882415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.923613] LNet: 30675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.955256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.078909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 760.143266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 760.190741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 760.232581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 760.281834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 760.327390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 760.380760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 760.449913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 760.541364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 760.614516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 760.670364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 760.725826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 760.791755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 760.854815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 760.926287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 760.987848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 761.049289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 761.100911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 761.182304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 761.221438] LNetError: 30725:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 770.199569] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:28:36 (1785774516) [ 770.311850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.352632] LNet: 31290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.395381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.575988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 779.927745] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:28:45 (1785774525) [ 781.743812] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg118-server [ 783.487445] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:28:50 (1785774530) [ 783.842286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.875199] LNet: 32228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.908687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.969772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 784.010298] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 785.270669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[19-23]@tcp [ 785.298833] LNet: 32362:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 785.353913] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.20@tcp is being used as a gateway but routing feature is not turned on [ 785.381994] LNetError: 32232:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.20@tcp has gone from up to down [ 786.357389] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.21@tcp is being used as a gateway but routing feature is not turned on [ 786.376049] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 6 previous similar messages [ 787.403057] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.21@tcp is being used as a gateway but routing feature is not turned on [ 787.424866] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 789.441541] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.20@tcp is being used as a gateway but routing feature is not turned on [ 789.445742] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 793.312169] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:28:59 (1785774539) [ 793.529660] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.20@tcp is being used as a gateway but routing feature is not turned on [ 793.539051] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 795.403241] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg118-server [ 797.546487] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:29:03 (1785774543) [ 797.667254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.624458] LNetError: 32231:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.201.20@tcp [ 836.576141] LNetError: 32236:0:(socklnd_cb.c:1675:ksocknal_recv_hello()) Error -11 reading HELLO from 192.168.201.19 [ 836.587303] LNetError: Unexpected error -11 connecting to 192.168.201.19@tcp at host 192.168.201.19:988 [ 836.642364] LNetError: 32232:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.19@tcp has gone from up to down [ 836.665396] LNetError: 32232:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 836.686714] LNet: 33292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.731855] LNetError: 33292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 836.770086] LNet: Removed LNI 192.168.201.18@tcp [ 836.867754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.979442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 837.038146] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 838.354570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[19-24/2]@tcp [ 838.401940] LNet: 33422:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 838.411467] LNet: 33422:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 838.440187] LNetError: 33308:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.21@tcp is being used as a gateway but routing feature is not turned on [ 838.446123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[19-24/2]@tcp [ 838.455242] LNetError: 33308:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 838.456449] LNetError: 33308:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.21@tcp has gone from up to down [ 849.388404] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:29:55 (1785774595) [ 849.586821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.824241] LNetError: 33313:0:(socklnd_cb.c:1675:ksocknal_recv_hello()) Error -11 reading HELLO from 192.168.201.19 [ 889.843481] LNetError: Unexpected error -11 connecting to 192.168.201.19@tcp at host 192.168.201.19:988 [ 890.601815] LNet: 34014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.611960] LNetError: 34014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.641292] LNet: Removed LNI 192.168.201.18@tcp [ 890.736316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.954964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 891.132820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.173143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.259216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 891.329131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.380669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.486674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 891.615259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.674699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.741357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 891.902630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.974626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.087803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 892.181816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.235741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.335844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 892.455904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.502430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.598681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 892.722547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.782938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.866797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 892.985114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.065911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.132698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 893.244324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.313166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.388621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 893.504653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.554141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.627855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 893.693421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.734839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.812293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 901.842404] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:30:48 (1785774648) [ 903.916846] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg118-server [ 906.258390] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:30:52 (1785774652) [ 909.021169] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg118-server [ 910.739780] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:30:57 (1785774657) [ 910.988848] LNet: 35424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.993672] LNet: 35424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 911.607317] Key type .llcrypt unregistered [ 911.609518] Key type ._llcrypt unregistered [ 920.678021] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 920.794513] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 920.806261] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 920.812566] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 922.503430] Key type ._llcrypt registered [ 922.505341] Key type .llcrypt registered [ 922.734879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 922.752754] alg: No test for adler32 (adler32-zlib) [ 923.806581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.874317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 923.916103] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 924.871973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 924.889977] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 927.435799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 927.454569] LNet: Removed LNI 10.1.2.3@tcp [ 927.555334] LNet: 36133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.560269] LNetError: 36133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.572849] LNet: Removed LNI 192.168.201.18@tcp [ 928.471807] Key type .llcrypt unregistered [ 928.477589] Key type ._llcrypt unregistered [ 944.490586] Key type ._llcrypt registered [ 944.491708] Key type .llcrypt registered [ 946.294299] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:31:32 (1785774692) [ 947.082436] Key type .llcrypt unregistered [ 947.084645] Key type ._llcrypt unregistered [ 956.299736] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 956.633878] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 956.644939] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 956.651650] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 959.577060] Key type ._llcrypt registered [ 959.579071] Key type .llcrypt registered [ 959.746165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 959.759305] alg: No test for adler32 (adler32-zlib) [ 960.873146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.975576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 961.011170] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 961.942257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 961.959973] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 964.655228] LNet: 37623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.667594] LNetError: 37623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 964.679241] LNet: Removed LNI 10.1.2.3@tcp [ 964.693723] LNet: Removed LNI 192.168.201.18@tcp [ 965.502183] Key type .llcrypt unregistered [ 965.505672] Key type ._llcrypt unregistered [ 981.459234] Key type ._llcrypt registered [ 981.463399] Key type .llcrypt registered [ 983.024933] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:32:09 (1785774729) [ 983.821244] Key type .llcrypt unregistered [ 983.829567] Key type ._llcrypt unregistered [ 992.990385] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 993.091664] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 993.105761] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 993.153109] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 995.304837] Key type ._llcrypt registered [ 995.308837] Key type .llcrypt registered [ 995.609175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 995.624288] alg: No test for adler32 (adler32-zlib) [ 996.651685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 996.711687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 996.744717] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 1000.770623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1000.832305] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1008.291550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 1008.326952] LNet: Removed LNI 10.1.2.3@tcp [ 1008.388924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 1008.483699] LNet: Removed LNI 10.1.2.31@tcp [ 1008.519818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1008.585832] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1014.689069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 1014.720110] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 1016.415875] LNet: 39543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.440389] LNetError: 39543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 1016.459191] LNet: Removed LNI 10.1.2.31@tcp [ 1016.512343] LNet: Removed LNI 10.1.2.3@tcp [ 1018.770349] Key type .llcrypt unregistered [ 1018.772094] Key type ._llcrypt unregistered [ 1035.796423] Key type ._llcrypt registered [ 1035.799381] Key type .llcrypt registered [ 1037.772200] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:33:03 (1785774783) [ 1038.347663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1038.381426] alg: No test for adler32 (adler32-zlib) [ 1039.481968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1039.607739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1039.686980] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1040.845954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1040.899520] LNet: 40497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.915905] LNetError: 40497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1040.966075] LNet: Removed LNI 192.168.201.18@tcp [ 1041.018636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.120069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.18@tcp [ 1041.172951] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1042.417464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.18@tcp [ 1042.479594] LNet: Removed LNI 192.168.201.18@tcp [ 1042.582964] LNet: 40633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.546214] Key type .llcrypt unregistered [ 1043.547952] Key type ._llcrypt unregistered [ 1057.814405] Key type ._llcrypt registered [ 1057.817228] Key type .llcrypt registered [ 1059.544363] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:33:25 (1785774805) [ 1061.682756] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg118-server [ 1064.055794] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:33:30 (1785774810) [ 1064.699418] Key type .llcrypt unregistered [ 1064.707869] Key type ._llcrypt unregistered [ 1071.563360] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1071.772522] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1071.790138] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1071.800969] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1074.861495] Key type ._llcrypt registered [ 1074.865208] Key type .llcrypt registered [ 1075.000600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1075.007720] alg: No test for adler32 (adler32-zlib) [ 1075.911047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1075.977333] LNet: Added LNI 192.168.201.18@tcp1 [8/256/0/180] [ 1076.594825] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 1079.274154] LNet: 42472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1079.280255] LNetError: 42472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1079.288145] LNet: Removed LNI 192.168.201.18@tcp1 [ 1079.315355] LNet: Removed LNI 10.1.2.3@tcp2 [ 1080.045018] Key type .llcrypt unregistered [ 1080.048183] Key type ._llcrypt unregistered [ 1095.037260] Key type ._llcrypt registered [ 1095.043273] Key type .llcrypt registered [ 1096.420974] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:34:02 (1785774842) [ 1096.931352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1096.958110] alg: No test for adler32 (adler32-zlib) [ 1097.874937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1097.911320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1097.931366] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1098.915632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1098.969872] LNet: 43444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1098.984872] LNetError: 43444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1099.022745] LNet: Removed LNI 192.168.201.18@tcp [ 1099.051210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1099.140657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 1099.180486] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1108.789536] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:34:14 (1785774854) [ 1108.966211] LNet: 44029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.003243] LNetError: 44029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1109.030274] LNet: Removed LNI 192.168.201.18@tcp [ 1110.135456] Key type .llcrypt unregistered [ 1110.136790] Key type ._llcrypt unregistered [ 1118.358452] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1118.530029] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1118.540851] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1118.561723] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1121.347454] Key type ._llcrypt registered [ 1121.352255] Key type .llcrypt registered [ 1121.611247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1121.633888] alg: No test for adler32 (adler32-zlib) [ 1122.755923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1122.862524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1122.920233] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1123.833838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1123.848743] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1126.676390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1126.700672] LNet: 44776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1126.713522] LNetError: 44776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1126.721424] LNet: Removed LNI 10.1.2.3@tcp [ 1126.734960] LNet: Removed LNI 192.168.201.18@tcp [ 1126.801377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1126.922295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1126.988242] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1127.000078] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1128.207120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1128.244452] LNet: 44804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.255119] LNetError: 44804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1128.270795] LNetError: 44804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1128.289475] LNet: Removed LNI 10.1.2.3@tcp [ 1128.314403] LNet: Removed LNI 192.168.201.18@tcp [ 1128.407543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1128.559570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1128.608114] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1128.630220] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1129.501892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1129.563807] LNet: Added LNI 192.168.201.18@tcp2 [8/256/0/180] [ 1131.210752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1131.225927] LNet: 44966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1131.230034] LNetError: 44966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1131.234504] LNetError: 44966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1131.240882] LNet: Removed LNI 10.1.2.3@tcp [ 1131.242824] LNet: Removed LNI 192.168.201.18@tcp [ 1131.251764] LNet: Removed LNI 192.168.201.18@tcp2 [ 1131.266719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1131.316471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1131.336381] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1131.340062] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1131.343687] LNet: Added LNI 192.168.201.18@tcp2 [8/256/0/180] [ 1142.261889] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:34:47 (1785774887) [ 1142.344508] LNet: 45554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1142.352392] LNetError: 45554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1142.363194] LNetError: 45554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1142.379923] LNet: Removed LNI 10.1.2.3@tcp [ 1142.385590] LNet: Removed LNI 192.168.201.18@tcp [ 1142.435273] LNet: Removed LNI 192.168.201.18@tcp2 [ 1143.368033] Key type .llcrypt unregistered [ 1143.376120] Key type ._llcrypt unregistered [ 1151.284180] Key type ._llcrypt registered [ 1151.285779] Key type .llcrypt registered [ 1151.634995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1151.671266] alg: No test for adler32 (adler32-zlib) [ 1152.849457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1152.961440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1153.002454] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1154.179475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1154.206782] LNet: 45960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.219239] LNetError: 45960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1154.243168] LNet: Removed LNI 192.168.201.18@tcp [ 1154.263992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1154.374933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1154.415236] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1155.224767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1155.265667] LNet: 45990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.283601] LNetError: 45990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.301851] LNet: Removed LNI 192.168.201.18@tcp [ 1155.333633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1155.480442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1155.536643] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1156.141284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1156.163719] LNet: Removed LNI 192.168.201.18@tcp [ 1156.182865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1156.252580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1156.269460] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1156.808077] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1156.986773] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1157.003614] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1157.011694] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1159.583651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1159.623460] LNet: 46098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1159.632788] LNet: 46098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1159.651189] LNetError: 46098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1159.653370] LNetError: 46098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1159.688846] LNet: Removed LNI 192.168.201.18@tcp [ 1159.719598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1159.843463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1159.949141] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1169.543286] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:35:15 (1785774915) [ 1169.677784] LNet: 46688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1169.691442] LNetError: 46688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1169.724273] LNet: Removed LNI 192.168.201.18@tcp [ 1170.746824] Key type .llcrypt unregistered [ 1170.751085] Key type ._llcrypt unregistered [ 1178.774675] Key type ._llcrypt registered [ 1178.777589] Key type .llcrypt registered [ 1178.997935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1179.020226] alg: No test for adler32 (adler32-zlib) [ 1179.963906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1180.046979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1180.079531] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1181.287980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1181.364649] LNet: 47095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1181.389081] LNetError: 47095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1181.419897] LNet: Removed LNI 192.168.201.18@tcp [ 1181.458216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1181.591510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1181.635494] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1189.983747] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:35:36 (1785774936) [ 1190.055245] LNet: 47678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1190.084346] LNetError: 47678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1190.121259] LNet: Removed LNI 192.168.201.18@tcp [ 1191.078817] Key type .llcrypt unregistered [ 1191.080820] Key type ._llcrypt unregistered [ 1199.919630] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1200.038993] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1200.054311] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1200.070507] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1203.040494] Key type ._llcrypt registered [ 1203.043710] Key type .llcrypt registered [ 1203.355340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1203.383429] alg: No test for adler32 (adler32-zlib) [ 1204.418637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1204.509536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1204.568290] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1205.874606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1205.912265] LNet: 48141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1205.919797] LNetError: 48141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1205.945784] LNet: Removed LNI 192.168.201.18@tcp [ 1205.979929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1206.098995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1206.146174] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1214.593940] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:36:00 (1785774960) [ 1214.840128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1214.885412] LNet: 48739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1214.902897] LNetError: 48739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1214.941126] LNet: Removed LNI 192.168.201.18@tcp [ 1214.974811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1215.108369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1223.410787] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:36:09 (1785774969) [ 1223.616263] LNet: 49317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1224.675860] Key type .llcrypt unregistered [ 1224.679413] Key type ._llcrypt unregistered [ 1237.761596] Key type ._llcrypt registered [ 1237.766748] Key type .llcrypt registered [ 1237.931319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1237.943685] alg: No test for adler32 (adler32-zlib) [ 1238.879874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1238.916673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1238.933370] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1240.626855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1240.686584] LNet: 49721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.707910] LNetError: 49721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1240.731563] LNet: Removed LNI 192.168.201.18@tcp [ 1240.808481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1240.963794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1241.067525] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1250.496535] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:36:37 (1785774997) [ 1250.539659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1250.552930] LNet: 50306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1250.556304] LNetError: 50306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1250.565889] LNet: Removed LNI 192.168.201.18@tcp [ 1250.579805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1250.638441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1259.317060] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:36:45 (1785775005) [ 1261.645532] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1264.080442] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:36:50 (1785775010) [ 1265.605415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1267.625469] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:36:54 (1785775014) [ 1269.258244] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1271.813645] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:36:57 (1785775017) [ 1273.576491] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1276.015200] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:37:02 (1785775022) [ 1278.350796] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1281.049900] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:37:06 (1785775026) [ 1281.174884] LNet: 52359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1282.171348] Key type .llcrypt unregistered [ 1282.174773] Key type ._llcrypt unregistered [ 1291.644762] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1291.792566] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1291.799951] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1291.806177] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1294.770666] Key type ._llcrypt registered [ 1294.774885] Key type .llcrypt registered [ 1295.096726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1295.117316] alg: No test for adler32 (adler32-zlib) [ 1295.994420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1296.067610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1296.096946] LNet: Added LNI 192.168.201.18@tcp [8/512/0/180] [ 1296.105680] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1296.109532] LNet: 52690:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1296.285825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1296.323781] LNet: 52709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1296.343224] LNetError: 52709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1296.356385] LNet: Removed LNI 10.1.2.3@tcp [ 1296.360794] LNetError: 52709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.396347] LNet: Removed LNI 192.168.201.18@tcp [ 1296.423797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1296.559298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1296.606652] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1296.692242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1296.739059] LNet: 52733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1296.747965] LNetError: 52733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.771901] LNet: Removed LNI 192.168.201.18@tcp [ 1296.824897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1296.934385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1296.988366] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1296.998657] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1297.010115] LNet: 52741:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1297.184899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1297.217721] LNet: 52759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.222867] LNetError: 52759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1297.231859] LNet: Removed LNI 10.1.2.3@tcp [ 1297.233529] LNetError: 52759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1297.245203] LNet: Removed LNI 192.168.201.18@tcp [ 1297.291786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1297.422167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1297.477082] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1297.498167] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1297.503627] LNet: 52767:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1297.712669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1297.727143] LNet: 52793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.730910] LNetError: 52793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1297.733852] LNet: Removed LNI 10.1.2.3@tcp [ 1297.734928] LNetError: 52793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1297.749788] LNet: Removed LNI 192.168.201.18@tcp [ 1297.804102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1297.894870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1297.929419] LNet: Added LNI 192.168.201.18@tcp [8/256/0/182] [ 1297.932662] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1298.123203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1298.181541] LNet: 52821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.195390] LNetError: 52821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1298.201290] LNet: Removed LNI 10.1.2.3@tcp [ 1298.204698] LNetError: 52821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.217607] LNet: Removed LNI 192.168.201.18@tcp [ 1298.262789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1298.377490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1298.415774] LNet: Added LNI 192.168.201.18@tcp [8/256/0/182] [ 1298.435634] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1298.440089] LNet: 52829:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1298.581311] LNet: Removed LNI 10.1.2.3@tcp [ 1298.595589] LNet: Removed LNI 192.168.201.18@tcp [ 1299.557871] Key type .llcrypt unregistered [ 1299.560960] Key type ._llcrypt unregistered [ 1316.628980] Key type ._llcrypt registered [ 1316.631877] Key type .llcrypt registered [ 1317.886338] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:37:44 (1785775064) [ 1318.997447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1319.045402] alg: No test for adler32 (adler32-zlib) [ 1320.127407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1320.266954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1320.307366] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1321.692741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.18 [ 1321.728312] LNet: 53806:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1321.745197] LNet: Removed LNI 192.168.201.18@tcp [ 1330.013560] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:37:56 (1785775076) [ 1330.288586] LNet: 54376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1330.307614] LNetError: 54376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1330.332641] LNet: Removed LNI 192.168.201.18@tcp [ 1331.028514] Key type .llcrypt unregistered [ 1331.031363] Key type ._llcrypt unregistered [ 1341.007413] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1341.338868] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1344.582329] Key type ._llcrypt registered [ 1344.583475] Key type .llcrypt registered [ 1344.777449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1344.790368] alg: No test for adler32 (adler32-zlib) [ 1345.982486] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1351.759922] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:38:18 (1785775098) [ 1352.058470] LNet: 55288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1352.063938] LNetError: 55288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1352.098531] LNet: Removed LNI 10.1.2.3@tcp [ 1352.566879] Key type .llcrypt unregistered [ 1352.569073] Key type ._llcrypt unregistered [ 1361.221806] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1361.586147] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1361.594937] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1361.612744] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1364.686244] Key type ._llcrypt registered [ 1364.698848] Key type .llcrypt registered [ 1364.902146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1364.917895] alg: No test for adler32 (adler32-zlib) [ 1365.851977] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1372.424896] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:38:38 (1785775118) [ 1374.244586] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1377.368804] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:38:43 (1785775123) [ 1377.542435] LNet: 56469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.553970] LNetError: 56469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1377.593378] LNet: Removed LNI 10.1.2.3@tcp [ 1379.188356] Key type .llcrypt unregistered [ 1379.189687] Key type ._llcrypt unregistered [ 1388.456039] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1388.758745] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1388.765098] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1388.776921] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1390.621432] Key type ._llcrypt registered [ 1390.623076] Key type .llcrypt registered [ 1390.976486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1391.015635] alg: No test for adler32 (adler32-zlib) [ 1392.052845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1392.178949] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1402.445682] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:39:09 (1785775149) [ 1402.923457] LNet: 57701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1402.927436] LNetError: 57701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1402.947146] LNet: Removed LNI 10.1.2.3@tcp [ 1403.470255] Key type .llcrypt unregistered [ 1403.471377] Key type ._llcrypt unregistered [ 1412.177194] Key type ._llcrypt registered [ 1412.183254] Key type .llcrypt registered [ 1412.551666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1412.588662] alg: No test for adler32 (adler32-zlib) [ 1413.506128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1413.532066] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1423.481306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.118@tcp [ 1424.213921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_interrupt -r 1 [ 1424.274658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_interrupt -r 1 [ 1424.316632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1424.335650] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1424.556841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1424.581876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1424.609819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1424.636805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1424.665864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1424.690498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1424.718051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1424.741404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1424.754762] LNet: 58141:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1424.772943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1424.797683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1424.803541] LNet: 1 local NIs in recovery (showing 1): 192.168.201.18@tcp [ 1424.830800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1424.862874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1424.904675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1424.935933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1424.968872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1425.541930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_dropped -r 1 [ 1425.612572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_dropped -r 1 [ 1425.679776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1425.711081] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1426.064546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1426.105872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1426.155541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1426.221691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1426.259266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1426.309470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1426.352593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1426.399964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1426.439419] LNet: 58250:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1426.473806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1426.517902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1426.569457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1426.609403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1426.656606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1426.693613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1426.745693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1427.355552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_aborted -r 1 [ 1427.421562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_aborted -r 1 [ 1427.470625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1427.526361] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1427.528914] LNet: Skipped 1 previous similar message [ 1427.779798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1427.832901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1427.882288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1427.935869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1427.994063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1428.024262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1428.074190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1428.112328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1428.129382] LNet: 58359:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1428.145439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1428.193562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1428.223762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1428.251089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1428.292840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1428.332058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1428.369638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1429.046551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_no_route -r 1 [ 1429.092967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_no_route -r 1 [ 1429.158521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1429.492773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1429.535852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1429.581400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1429.625529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1429.688686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1429.726507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1429.805890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1429.844502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1429.888202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1429.950313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1430.072761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1430.160235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1430.218239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1430.358819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1430.458332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1431.217546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_timeout -r 1 [ 1431.288263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_timeout -r 1 [ 1431.376774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1431.419105] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1431.425390] LNet: Skipped 1 previous similar message [ 1431.724498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1431.781107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1431.833546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1431.865855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1431.921339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1431.984316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1432.040125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1432.083587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1432.112481] LNet: 58578:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1432.120578] LNet: 58578:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1432.141709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1432.214817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1432.272956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1432.335496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1432.384372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1432.446071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1432.498746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1433.237184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_error -r 1 [ 1433.288633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_error -r 1 [ 1433.346730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1433.720936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1433.773114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1433.803483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1433.835928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1433.891947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1433.947482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1434.016938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1434.087175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1434.153365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1434.213910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1434.265455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1434.323374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1434.370965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1434.428681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1434.489918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1434.995861] LNet: 58728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1435.008279] LNetError: 58728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1435.023831] LNet: Removed LNI 192.168.201.18@tcp [ 1435.731563] Key type .llcrypt unregistered [ 1435.735199] Key type ._llcrypt unregistered [ 1456.627610] Key type ._llcrypt registered [ 1456.630717] Key type .llcrypt registered [ 1457.983787] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:40:04 (1785775204) [ 1459.007338] Key type .llcrypt unregistered [ 1459.010536] Key type ._llcrypt unregistered [ 1466.420851] Key type ._llcrypt registered [ 1466.426403] Key type .llcrypt registered [ 1466.741603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1466.772232] alg: No test for adler32 (adler32-zlib) [ 1467.731737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1467.776190] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1476.840713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.118@tcp [ 1489.477761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1489.557756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1489.597754] LNet: Added LNI 192.168.201.18@tcp1 [8/256/0/180] [ 1491.196412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_interrupt -r 1 [ 1491.239795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp1 -m GET -e local_interrupt -r 1 [ 1491.283465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_interrupt -r 1 [ 1491.362356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -m GET -e local_interrupt -r 1 [ 1491.429579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp -m GET -e local_interrupt -r 1 [ 1491.478611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp1 -m GET -e local_interrupt -r 1 [ 1491.594316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -m GET -e local_interrupt -r 1 [ 1491.679811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -m GET -e local_interrupt -r 1 [ 1491.741314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1491.799136] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1491.814769] LNet: 1 local NIs in recovery (showing 1): 192.168.201.18@tcp [ 1492.136788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1492.175053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1492.232808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1492.267781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1492.327504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1492.374696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1492.409155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1492.436354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1492.452987] LNet: 60199:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1492.476910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1492.502757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1492.534969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1492.564477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1492.587310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1492.614404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1492.644343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1493.397671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_dropped -r 1 [ 1493.456891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp1 -m GET -e local_dropped -r 1 [ 1493.510387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_dropped -r 1 [ 1493.583585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -m GET -e local_dropped -r 1 [ 1493.673322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp -m GET -e local_dropped -r 1 [ 1493.783861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp1 -m GET -e local_dropped -r 1 [ 1493.859578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -m GET -e local_dropped -r 1 [ 1493.917162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -m GET -e local_dropped -r 1 [ 1493.976753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1494.006958] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1494.014835] LNet: Skipped 2 previous similar messages [ 1494.386406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1494.453709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1494.519494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1494.575188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1494.622919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1494.670280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1494.725195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1494.783133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1494.816200] LNet: 60320:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1494.844635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1494.893692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1494.932866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1494.996505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1495.062633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1495.080655] LNet: There was an unexpected network error while writing to 192.168.201.18: rc = -22 [ 1495.083207] LNet: Skipped 2 previous similar messages [ 1495.140264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1495.208971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1496.109884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_aborted -r 1 [ 1496.147294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp1 -m GET -e local_aborted -r 1 [ 1496.189895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_aborted -r 1 [ 1496.229862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -m GET -e local_aborted -r 1 [ 1496.277359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp -m GET -e local_aborted -r 1 [ 1496.311110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp1 -m GET -e local_aborted -r 1 [ 1496.347626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -m GET -e local_aborted -r 1 [ 1496.395645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -m GET -e local_aborted -r 1 [ 1496.456962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1496.848826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1496.942152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1497.051287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1497.127939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1497.207785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1497.298810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1497.371708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1497.487222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1497.541347] LNet: 60443:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1497.585804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1497.704135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1497.804108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1497.864942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1497.968869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1498.054911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1498.137653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1499.236731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_no_route -r 1 [ 1499.312486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp1 -m GET -e local_no_route -r 1 [ 1499.385243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_no_route -r 1 [ 1499.476551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -m GET -e local_no_route -r 1 [ 1499.582062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp -m GET -e local_no_route -r 1 [ 1499.672235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp1 -m GET -e local_no_route -r 1 [ 1499.742882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -m GET -e local_no_route -r 1 [ 1499.833806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -m GET -e local_no_route -r 1 [ 1500.018836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1500.083184] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1500.092341] LNet: Skipped 4 previous similar messages [ 1500.779897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1500.869389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1500.930492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1500.975950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1501.038446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1501.105906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1501.164549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1501.240725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1501.265786] LNet: 60565:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1501.345019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1501.407556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1501.485142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1501.560858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1501.608271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1501.682893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1501.733294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1502.793717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_timeout -r 1 [ 1502.998311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp1 -m GET -e local_timeout -r 1 [ 1503.242707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_timeout -r 1 [ 1503.379103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -m GET -e local_timeout -r 1 [ 1503.522330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp -m GET -e local_timeout -r 1 [ 1503.664773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp1 -m GET -e local_timeout -r 1 [ 1503.778514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -m GET -e local_timeout -r 1 [ 1503.872338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -m GET -e local_timeout -r 1 [ 1503.998646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1504.742283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1504.810170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1504.884260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1505.021425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1505.111147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1505.177019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1505.232753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1505.280463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1505.317592] LNet: 60686:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1505.360345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1505.423378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1505.481207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1505.540842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1505.583960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1505.649017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1505.681453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1506.236276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_error -r 1 [ 1506.261463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp1 -m GET -e local_error -r 1 [ 1506.294453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_error -r 1 [ 1506.321029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -m GET -e local_error -r 1 [ 1506.344648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp -m GET -e local_error -r 1 [ 1506.387792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp1 -m GET -e local_error -r 1 [ 1506.433894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -m GET -e local_error -r 1 [ 1506.512298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -m GET -e local_error -r 1 [ 1506.664531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1506.711642] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1506.718850] LNet: Skipped 5 previous similar messages [ 1507.034214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1507.093409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1507.138804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1507.188936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1507.243461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1507.266578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1507.311599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1507.352644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1507.406950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1507.492775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1507.541934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1507.612718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1507.667616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1507.736803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1507.809044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1517.940827] LNet: 60885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.963423] LNetError: 60885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.975288] LNet: Removed LNI 192.168.201.18@tcp [ 1519.082253] LNet: Removed LNI 192.168.201.18@tcp1 [ 1519.953981] Key type .llcrypt unregistered [ 1519.963356] Key type ._llcrypt unregistered [ 1552.340896] Key type ._llcrypt registered [ 1552.345437] Key type .llcrypt registered [ 1554.499720] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:41:40 (1785775300) [ 1555.596981] Key type .llcrypt unregistered [ 1555.605367] Key type ._llcrypt unregistered [ 1565.633861] Key type ._llcrypt registered [ 1565.639824] Key type .llcrypt registered [ 1566.097603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1566.121551] alg: No test for adler32 (adler32-zlib) [ 1567.132054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1567.176048] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1576.419562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.118@tcp [ 1577.125713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e remote_dropped -r 1 [ 1577.155446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e remote_dropped -r 1 [ 1577.197608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1577.224589] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1577.531850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1577.559717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1577.587714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1577.632743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1577.710358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1577.755980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1577.797748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1577.841866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1577.860171] LNet: 62138:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1577.879340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1577.906954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1577.939476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1577.985834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1578.026872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1578.065655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1578.105501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1578.638690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e remote_error -r 1 [ 1578.669372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e remote_error -r 1 [ 1578.695411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1578.714811] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1578.943635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1578.967573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1578.990361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1579.029692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1579.059223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1579.094694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1579.135590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1579.165868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1579.190703] LNet: 62247:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1579.218534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1579.285981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1579.349309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1579.405298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1579.442170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1579.484643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1579.523406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1580.126519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e remote_timeout -r 1 [ 1580.192466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e remote_timeout -r 1 [ 1580.278430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1580.338851] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1580.766282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1580.812804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1580.852608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1580.893869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1580.952541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1580.998204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1581.051669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1581.128359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1581.156663] LNet: 62357:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1581.176463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1581.237707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1581.278435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1581.325600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1581.368567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1581.420519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1581.479491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1581.882182] LNet: 62398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1581.895623] LNetError: 62398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1581.961156] LNet: Removed LNI 192.168.201.18@tcp [ 1582.710502] Key type .llcrypt unregistered [ 1582.712776] Key type ._llcrypt unregistered [ 1606.221682] Key type ._llcrypt registered [ 1606.226749] Key type .llcrypt registered [ 1607.637889] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:42:34 (1785775354) [ 1608.509260] Key type .llcrypt unregistered [ 1608.512464] Key type ._llcrypt unregistered [ 1615.182345] Key type ._llcrypt registered [ 1615.190960] Key type .llcrypt registered [ 1615.578522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1615.626453] alg: No test for adler32 (adler32-zlib) [ 1616.628151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1616.674150] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1625.970543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.118@tcp [ 1637.822326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.932478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1637.969863] LNet: Added LNI 192.168.201.18@tcp1 [8/256/0/180] [ 1640.696945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e remote_dropped -r 1 [ 1640.767616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp1 -m GET -e remote_dropped -r 1 [ 1640.887437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e remote_dropped -r 1 [ 1640.982991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -m GET -e remote_dropped -r 1 [ 1641.063935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp -m GET -e remote_dropped -r 1 [ 1641.119965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp1 -m GET -e remote_dropped -r 1 [ 1641.194640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -m GET -e remote_dropped -r 1 [ 1641.293712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -m GET -e remote_dropped -r 1 [ 1641.387283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1641.422266] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1641.442246] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.118@tcp [ 1641.761684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1641.802160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1641.861767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1641.936319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1641.975090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1642.021609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1642.069954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1642.125794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1642.142132] LNet: 63888:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1642.172768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1642.232360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1642.302376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1642.360713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1642.435174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1642.501864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1642.572478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1642.628396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1643.707426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e remote_error -r 1 [ 1643.776065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp1 -m GET -e remote_error -r 1 [ 1643.849874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e remote_error -r 1 [ 1643.916911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -m GET -e remote_error -r 1 [ 1644.043889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp -m GET -e remote_error -r 1 [ 1644.142381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp1 -m GET -e remote_error -r 1 [ 1644.253976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -m GET -e remote_error -r 1 [ 1644.322362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -m GET -e remote_error -r 1 [ 1644.378787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1644.413689] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1644.418081] LNet: Skipped 2 previous similar messages [ 1644.821101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1644.882455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1644.930688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1645.001352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1645.103763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1645.184337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1645.235493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1645.307066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1645.352371] LNet: 64011:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1645.398178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1645.499321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1645.547675] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1645.642522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1645.740562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1645.837711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1645.914651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1645.967219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1646.027673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1647.601783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e remote_timeout -r 1 [ 1647.795796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp1 -m GET -e remote_timeout -r 1 [ 1647.914353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e remote_timeout -r 1 [ 1647.993510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -m GET -e remote_timeout -r 1 [ 1648.158551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp -m GET -e remote_timeout -r 1 [ 1648.276468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp1 -m GET -e remote_timeout -r 1 [ 1648.416560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -m GET -e remote_timeout -r 1 [ 1648.530357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -m GET -e remote_timeout -r 1 [ 1648.622272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.118@tcp [ 1648.658459] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1649.293068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1649.434815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1649.511674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1649.607386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1649.702277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1649.825492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1649.902501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1649.974921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1650.017912] LNet: 64135:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1650.061324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1650.136447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1650.223443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1650.337266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1650.437286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1650.517413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1650.643332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1650.709152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1659.035726] LNet: 64214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1659.039488] LNetError: 64214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1659.046069] LNet: Removed LNI 192.168.201.18@tcp [ 1659.067480] LNet: Removed LNI 192.168.201.18@tcp1 [ 1659.413524] Key type .llcrypt unregistered [ 1659.416062] Key type ._llcrypt unregistered [ 1679.533218] Key type ._llcrypt registered [ 1679.539940] Key type .llcrypt registered [ 1681.020952] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:43:47 (1785775427) [ 1683.041879] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1685.217482] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:43:51 (1785775431) [ 1686.195641] Key type .llcrypt unregistered [ 1686.201528] Key type ._llcrypt unregistered [ 1695.498532] Key type ._llcrypt registered [ 1695.504117] Key type .llcrypt registered [ 1695.815547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1695.848195] alg: No test for adler32 (adler32-zlib) [ 1696.867302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1696.926550] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1706.809835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.118@tcp [ 1707.516855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e network_timeout -r 1 [ 1707.543358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e network_timeout -r 1 [ 1707.574385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.118@tcp [ 1707.591658] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1707.831100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1707.883253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1707.909088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1707.931920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1707.970151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1707.999938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1708.028929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1708.053100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1708.079929] LNet: 65763:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1708.103939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1708.147701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1708.192362] LNet: 1 local NIs in recovery (showing 1): 192.168.201.18@tcp [ 1708.199845] LNet: There was an unexpected network error while writing to 192.168.201.18: rc = -22 [ 1708.206800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1708.239956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1708.265413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1708.294651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1708.527752] LNet: 65802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.534386] LNetError: 65802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.551229] LNet: Removed LNI 192.168.201.18@tcp [ 1710.080534] Key type .llcrypt unregistered [ 1710.088543] Key type ._llcrypt unregistered [ 1731.819663] Key type ._llcrypt registered [ 1731.822923] Key type .llcrypt registered [ 1732.110725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1732.141262] alg: No test for adler32 (adler32-zlib) [ 1733.190304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1733.230745] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1744.168499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.118@tcp [ 1754.855841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.913559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1754.955577] LNet: Added LNI 192.168.201.18@tcp1 [8/256/0/180] [ 1756.069208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e network_timeout -r 1 [ 1756.094224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp1 -m GET -e network_timeout -r 1 [ 1756.120330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e network_timeout -r 1 [ 1756.160972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -m GET -e network_timeout -r 1 [ 1756.194589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp -m GET -e network_timeout -r 1 [ 1756.230548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp1 -m GET -e network_timeout -r 1 [ 1756.263690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -m GET -e network_timeout -r 1 [ 1756.294524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -m GET -e network_timeout -r 1 [ 1756.322956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.118@tcp [ 1756.366402] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1756.589901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1756.637248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1756.682520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1756.721416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1756.750706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1756.773656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1756.791435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1756.812297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1756.829266] LNet: 66683:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1756.832397] LNet: 1 local NIs in recovery (showing 1): 192.168.201.18@tcp [ 1756.848533] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.118@tcp [ 1756.855393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1756.879141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1756.900336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1756.925700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1756.949280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1756.975827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1762.572530] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1763.075667] LNet: 66756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.082779] LNetError: 66756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.103261] LNet: Removed LNI 192.168.201.18@tcp [ 1763.116185] LNet: Removed LNI 192.168.201.18@tcp1 [ 1763.888561] Key type .llcrypt unregistered [ 1763.893345] Key type ._llcrypt unregistered [ 1789.220857] Key type ._llcrypt registered [ 1789.222255] Key type .llcrypt registered [ 1790.635255] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:45:37 (1785775537) [ 1791.391323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1791.413857] alg: No test for adler32 (adler32-zlib) [ 1792.397610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.465319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1792.484781] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1793.684907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1793.726714] LNet: Added LNI 192.168.201.18@tcp1 [8/256/0/180] [ 1795.307224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 1795.428305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1795.512398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1795.574795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1795.643415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1795.720807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1795.773053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.18@tcp [ 1795.801785] LNetError: 67859:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.18@tcp to recovery [ 1796.513872] LNet: 1 local NIs in recovery (showing 1): 192.168.201.18@tcp [ 1796.515412] LNet: There was an unexpected network error while writing to 192.168.201.18: rc = -22 [ 1798.563209] LNet: There was an unexpected network error while writing to 192.168.201.18: rc = -22 [ 1799.412856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1802.656363] LNet: There was an unexpected network error while writing to 192.168.201.18: rc = -22 [ 1803.204121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1803.295333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1803.422609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.476420] LNet: 67967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.490996] LNetError: 67967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.512131] LNet: Removed LNI 192.168.201.18@tcp [ 1804.517207] LNetError: 67967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.540170] LNet: Removed LNI 192.168.201.18@tcp1 [ 1804.588943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.675895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1804.703469] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1805.899549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1805.920346] LNet: Added LNI 192.168.201.18@tcp1 [8/256/0/180] [ 1806.612245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 1806.645335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1806.716721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1806.775875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1806.829166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1806.861915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1806.913822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1806.974927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.18@tcp [ 1807.008612] LNetError: 68239:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.18@tcp to recovery [ 1807.712651] LNet: There was an unexpected network error while writing to 192.168.201.18: rc = -22 [ 1810.825183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1813.856434] LNet: There was an unexpected network error while writing to 192.168.201.18: rc = -22 [ 1813.865713] LNet: Skipped 1 previous similar message [ 1819.118936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1819.188887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1819.276467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1828.405082] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:46:14 (1785775574) [ 1828.531609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.566697] LNet: 68952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.579970] LNetError: 68952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1829.600132] LNet: Removed LNI 192.168.201.18@tcp [ 1829.626820] LNet: Removed LNI 192.168.201.18@tcp1 [ 1829.684922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.748863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1829.773920] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1830.731169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1830.754950] LNet: Added LNI 192.168.201.18@tcp1 [8/256/0/180] [ 1831.986695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 1832.094609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1832.144306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1832.188712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1832.228782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1832.268720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1832.307784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.18@tcp --health 0 [ 1832.452426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1832.801145] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.18@tcp [ 1833.825054] LNet: There was an unexpected network error while writing to 192.168.201.18: rc = -22 [ 1833.828054] LNet: Skipped 1 previous similar message [ 1848.300392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1848.491196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1848.557274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1848.617087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.18@tcp --health 500 [ 1848.821933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1854.815224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1854.906864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1854.944902] LNet: 69442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1854.955048] LNetError: 69442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1854.967219] LNetError: 69442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1854.979434] LNet: Removed LNI 192.168.201.18@tcp [ 1855.022068] LNet: Removed LNI 192.168.201.18@tcp1 [ 1855.062714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1855.147478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1855.183596] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1856.411052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1856.438782] LNet: Added LNI 192.168.201.18@tcp1 [8/256/0/180] [ 1857.739838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 1857.791869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1857.883725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1857.925790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1857.981916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1858.029367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1858.073585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1858.113221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.18@tcp --health 0 [ 1859.232411] LNet: There was an unexpected network error while writing to 192.168.201.18: rc = -22 [ 1859.249161] LNet: Skipped 3 previous similar messages [ 1859.620507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1870.471830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1870.539202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1870.612743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1881.162713] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:47:07 (1785775627) [ 1881.989336] LNet: 70490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.013233] LNetError: 70490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.028603] LNetError: 70490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1882.050115] LNet: Removed LNI 192.168.201.18@tcp [ 1882.064071] LNet: Removed LNI 192.168.201.18@tcp1 [ 1882.921769] Key type .llcrypt unregistered [ 1882.923549] Key type ._llcrypt unregistered [ 1893.024263] Key type ._llcrypt registered [ 1893.033539] Key type .llcrypt registered [ 1893.410486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1893.452048] alg: No test for adler32 (adler32-zlib) [ 1894.575432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1894.629434] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1906.698310] Lustre: DEBUG MARKER: Initial discovery [ 1906.997382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.118@tcp [ 1911.460906] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1911.580666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1911.628996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.118@tcp [ 1911.675979] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1911.714120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1911.763719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.118@tcp -r 1 -m GET -l 3 [ 1911.819408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.118@tcp -r 1 -m GET -e local_error [ 1911.871582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.118@tcp -r 1 -m PUT -l 6 [ 1912.032301] LNet: 1 local NIs in recovery (showing 1): 192.168.201.18@tcp [ 1913.510392] Lustre: DEBUG MARKER: Force oleg118-client.virtnet to discover 192.168.201.118@tcp (in background) [ 1913.837965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.118@tcp [ 1916.252343] Lustre: DEBUG MARKER: Force oleg118-server to discover 192.168.201.18@tcp [ 1916.963993] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1921.190360] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 1921.847122] Lustre: DEBUG MARKER: Wait for 71257 [ 1924.198259] Lustre: DEBUG MARKER: Finished wait on 71257 [ 1924.370526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1924.407050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1925.005468] LNet: 71706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1925.024274] LNetError: 71706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1925.043856] LNet: Removed LNI 192.168.201.18@tcp [ 1925.553729] Key type .llcrypt unregistered [ 1925.559103] Key type ._llcrypt unregistered [ 1945.643348] Key type ._llcrypt registered [ 1945.644528] Key type .llcrypt registered [ 1947.820258] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:48:13 (1785775693) [ 1948.722243] Key type .llcrypt unregistered [ 1948.726787] Key type ._llcrypt unregistered [ 1958.016239] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1958.228960] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1958.242532] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1958.251976] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1961.073234] Key type ._llcrypt registered [ 1961.075301] Key type .llcrypt registered [ 1961.226672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1961.243462] alg: No test for adler32 (adler32-zlib) [ 1962.132229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.166894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1962.180360] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1962.987086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1963.028292] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1972.940969] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:48:39 (1785775719) [ 1973.093038] LNet: 73779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.100835] LNetError: 73779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1973.108727] LNet: Removed LNI 10.1.2.3@tcp [ 1973.128846] LNet: Removed LNI 192.168.201.18@tcp [ 1973.598408] Key type .llcrypt unregistered [ 1973.600777] Key type ._llcrypt unregistered [ 1985.463775] Key type ._llcrypt registered [ 1985.469976] Key type .llcrypt registered [ 1985.823633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1985.862722] alg: No test for adler32 (adler32-zlib) [ 1987.040435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1987.132356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1987.205350] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 1988.964849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1989.002966] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1990.455947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.18@tcp [ 1991.521117] LNetError: Unexpected error -22 connecting to 192.168.201.18@tcp at host 192.168.201.18:988 [ 2000.195824] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:49:06 (1785775746) [ 2000.380815] LNet: 74915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.393810] LNetError: 74915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2000.403186] LNet: Removed LNI 10.1.2.3@tcp [ 2000.423038] LNet: Removed LNI 192.168.201.18@tcp [ 2001.509720] Key type .llcrypt unregistered [ 2001.511735] Key type ._llcrypt unregistered [ 2010.024321] Key type ._llcrypt registered [ 2010.026516] Key type .llcrypt registered [ 2010.408076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2010.458057] alg: No test for adler32 (adler32-zlib) [ 2011.457892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2011.512782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2011.550548] LNet: Added LNI 192.168.201.18@tcp1 [8/256/0/180] [ 2012.286360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2012.337624] LNet: Added LNI 192.168.201.18@tcp2 [8/256/0/180] [ 2013.580142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.18@tcp1 --nid 192.168.201.18@tcp2 [ 2018.047936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.18@tcp1 192.168.201.18@tcp2 [ 2018.060973] LNetError: 75758:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.18@tcp1 to 192.168.201.18@tcp2 - Route table may be misconfigured [ 2018.079796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.18@tcp2 192.168.201.18@tcp1 [ 2025.480916] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:49:31 (1785775771) [ 2025.728412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2025.777655] LNet: 76332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2025.786747] LNetError: 76332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2025.794313] LNet: Removed LNI 192.168.201.18@tcp1 [ 2026.858335] LNet: Removed LNI 192.168.201.18@tcp2 [ 2026.911943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2026.999787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2027.023599] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2028.224582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2028.265955] LNet: Added LNI 192.168.201.18@tcp1 [8/256/0/180] [ 2029.152613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2029.212331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.18@tcp 192.168.201.18@tcp [ 2029.261951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.18@tcp1 192.168.201.18@tcp1 [ 2029.324791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.18@tcp -d 192.168.201.18@tcp -e network_timeout [ 2029.405948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -e network_timeout [ 2029.475627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -e network_timeout [ 2029.538391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -e network_timeout [ 2029.609315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2029.668519] LNet: There was an unexpected network error while writing to 192.168.201.18: rc = -22 [ 2029.699293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2029.852232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2029.985948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2030.051496] LNet: 1 local NIs in recovery (showing 1): 192.168.201.18@tcp [ 2036.901923] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:49:43 (1785775783) [ 2036.978809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2037.000866] LNet: 77187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.012819] LNetError: 77187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2037.019391] LNetError: 77187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2037.027856] LNet: Removed LNI 192.168.201.18@tcp [ 2038.056376] LNet: Removed LNI 192.168.201.18@tcp1 [ 2038.105475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2038.259637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 2038.510985] LNet: 77231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.939473] Key type .llcrypt unregistered [ 2038.943858] Key type ._llcrypt unregistered [ 2052.357542] Key type ._llcrypt registered [ 2052.358806] Key type .llcrypt registered [ 2053.909579] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:50:00 (1785775800) [ 2054.485268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2054.526738] alg: No test for adler32 (adler32-zlib) [ 2055.506687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2057.303565] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 2059.317450] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:50:05 (1785775805) [ 2059.587900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2059.642150] LNet: 78329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2059.696490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.794181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2059.869102] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2060.820520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2060.836786] LNet: Added LNI 192.168.201.18@tcp1 [8/256/0/180] [ 2062.077533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2062.150456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp1 [ 2062.210732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp1 [ 2069.338736] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:50:16 (1785775816) [ 2070.994294] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg118-server [ 2072.603587] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:50:19 (1785775819) [ 2074.898135] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg118-server [ 2077.007941] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:50:23 (1785775823) [ 2079.082650] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg118-server [ 2081.421198] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:50:27 (1785775827) [ 2083.417492] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg118-server [ 2086.107740] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:50:32 (1785775832) [ 2088.861843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg118-server [ 2091.072367] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:50:37 (1785775837) [ 2093.211640] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg118-server [ 2095.013948] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:50:41 (1785775841) [ 2096.783347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg118-server [ 2099.905613] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:50:45 (1785775845) [ 2103.457316] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg118-server [ 2106.760416] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:50:52 (1785775852) [ 2109.757706] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg118-server [ 2112.203048] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:50:58 (1785775858) [ 2112.592304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2112.613127] LNet: 82211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.618656] LNetError: 82211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.634818] LNet: Removed LNI 192.168.201.18@tcp [ 2113.636911] LNetError: 82211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.681675] LNet: Removed LNI 192.168.201.18@tcp1 [ 2113.699972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.766146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2113.788544] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2114.396458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 2114.522501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2114.810922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2114.833752] LNet: 82358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2114.839737] LNetError: 82358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2115.879461] LNet: Removed LNI 192.168.201.18@tcp [ 2115.949497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2116.022600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 2116.080560] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2116.931911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2117.121590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2117.151396] LNet: 82503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.157852] LNetError: 82503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2118.187028] LNet: Removed LNI 192.168.201.18@tcp [ 2118.246476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2118.336792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2118.378564] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2119.374883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 2119.475559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2119.687989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2120.742595] LNet: Removed LNI 192.168.201.18@tcp [ 2120.782244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2120.828861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 2120.878477] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2122.092603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2122.356546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2122.389131] LNet: 82796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2122.397208] LNet: 82796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2122.409144] LNetError: 82796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2122.419965] LNetError: 82796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2122.448694] LNet: Removed LNI 192.168.201.18@tcp [ 2122.470838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.550357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2122.594303] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2123.429846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 2123.497107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2123.705444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2124.774691] LNet: Removed LNI 192.168.201.18@tcp [ 2124.843777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2124.919823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 2124.953839] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2125.895384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2126.090093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.154562] LNet: Removed LNI 192.168.201.18@tcp [ 2127.190934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.261736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2127.339405] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2128.080620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 2128.192912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2128.545195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.639475] LNet: Removed LNI 192.168.201.18@tcp [ 2129.696634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.813257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 2129.852191] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2130.674797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2130.999251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.018460] LNet: 83382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.023618] LNet: 83382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2131.033219] LNetError: 83382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2131.041469] LNetError: 83382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2131.061175] LNet: Removed LNI 192.168.201.18@tcp [ 2131.095694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2131.166371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2131.198760] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2132.306290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 2132.464824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2134.019533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2135.146640] LNet: Removed LNI 192.168.201.18@tcp [ 2135.186499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.296621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 2135.360283] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2136.381538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2136.663165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.770564] LNet: Removed LNI 192.168.201.18@tcp [ 2137.796572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.879423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2137.929401] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2139.078961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 2139.234852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2139.666971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2140.774713] LNet: Removed LNI 192.168.201.18@tcp [ 2140.833976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.985124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 2141.054337] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2141.993023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2143.631235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.689181] LNet: Removed LNI 192.168.201.18@tcp [ 2144.728128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2144.842704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2144.868767] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2145.481486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 2145.520591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2145.538418] LNet: Removed LNI 192.168.201.18@tcp [ 2145.553132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.603459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2145.621535] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2146.307802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 2146.553804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2146.664634] LNet: Removed LNI 192.168.201.18@tcp [ 2146.702326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.782383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 2146.838843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2146.867414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.935730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 2147.017132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2147.029377] LNet: 84282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2147.033783] LNet: 84282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 2147.045875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2147.075694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 2147.126367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2147.149661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2147.258121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 2147.320701] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2148.566414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2148.592626] LNetError: 84427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.596432] LNetError: 84427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2148.629650] LNet: Removed LNI 192.168.201.18@tcp [ 2148.655154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.764534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2157.552663] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:51:43 (1785775903) [ 2157.706356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2157.817574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.052376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 2158.106627] LNet: Added LNI 192.168.201.18@tcp231 [8/256/0/180] [ 2160.672707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2160.778861] LNet: Removed LNI 192.168.201.18@tcp231 [ 2160.853510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2161.014245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2161.056443] LNet: Added LNI 192.168.201.18@tcp231 [8/256/0/180] [ 2161.063345] LNet: 85155:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2161.963389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2162.009419] LNet: Removed LNI 192.168.201.18@tcp231 [ 2162.040436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 2162.071558] LNet: Added LNI 192.168.201.18@tcp231 [8/256/0/180] [ 2164.100949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2164.164840] LNet: Removed LNI 192.168.201.18@tcp231 [ 2164.217132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2164.334164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2164.378291] LNet: Added LNI 192.168.201.18@tcp231 [8/256/0/0] [ 2164.386319] LNet: 85308:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2164.879783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2164.928134] LNet: Removed LNI 192.168.201.18@tcp231 [ 2164.945363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 2164.988293] LNet: Added LNI 192.168.201.18@tcp231 [8/256/0/0] [ 2167.306684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2167.352794] LNet: Removed LNI 192.168.201.18@tcp231 [ 2167.403175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2167.521849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2167.550361] LNet: Added LNI 192.168.201.18@tcp231 [8/256/0/60] [ 2167.555422] LNet: 85459:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2168.239777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2168.283285] LNet: Removed LNI 192.168.201.18@tcp231 [ 2168.308570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 2168.342854] LNet: Added LNI 192.168.201.18@tcp231 [8/256/0/60] [ 2170.147347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2170.201638] LNet: Removed LNI 192.168.201.18@tcp231 [ 2170.230107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.344285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2170.390387] LNet: Added LNI 192.168.201.18@tcp231 [8/256/0/180] [ 2170.396243] LNet: 85611:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2171.031910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2171.072671] LNet: Removed LNI 192.168.201.18@tcp231 [ 2171.088071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 2171.130793] LNet: Added LNI 192.168.201.18@tcp231 [8/256/0/180] [ 2180.802440] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:52:07 (1785775927) [ 2180.958819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2180.997775] LNet: 86315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.000216] LNet: 86315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 2181.020172] LNetError: 86315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2181.028251] LNetError: 86315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2181.060310] LNet: Removed LNI 192.168.201.18@tcp231 [ 2181.105159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.262359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 2181.316140] LNet: Added LNI 192.168.201.18@tcp232 [8/256/0/180] [ 2182.512348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2182.662904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2182.726805] LNet: Removed LNI 192.168.201.18@tcp232 [ 2182.772811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 2182.823420] LNet: Added LNI 192.168.201.18@tcp232 [8/256/0/180] [ 2184.302828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2184.344553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2184.396893] LNet: Removed LNI 192.168.201.18@tcp232 [ 2184.420425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 2184.486823] LNet: Added LNI 192.168.201.18@tcp232 [8/256/0/180] [ 2185.548359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2185.611056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2185.655239] LNet: Removed LNI 192.168.201.18@tcp232 [ 2185.674962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 2185.712735] LNet: Added LNI 192.168.201.18@tcp232 [8/256/0/180] [ 2186.842522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2186.919421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2186.975782] LNet: Removed LNI 192.168.201.18@tcp232 [ 2186.999227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 2195.611614] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:52:21 (1785775941) [ 2197.209255] Key type .llcrypt unregistered [ 2197.213268] Key type ._llcrypt unregistered [ 2204.835431] Key type ._llcrypt registered [ 2204.837036] Key type .llcrypt registered [ 2205.100632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2205.135715] alg: No test for adler32 (adler32-zlib) [ 2206.204267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2206.254769] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2214.318846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.118@tcp [ 2226.108768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2226.221523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2226.286261] LNet: Added LNI 192.168.201.18@tcp1 [8/256/0/180] [ 2229.458066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_interrupt -r 2 [ 2229.533872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp1 -m GET -e local_interrupt -r 2 [ 2229.618100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_interrupt -r 2 [ 2229.687093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -m GET -e local_interrupt -r 2 [ 2229.784493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp -m GET -e local_interrupt -r 2 [ 2229.834479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp1 -m GET -e local_interrupt -r 2 [ 2229.928764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -m GET -e local_interrupt -r 2 [ 2230.018913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -m GET -e local_interrupt -r 2 [ 2230.125853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.18@tcp 192.168.201.118@tcp [ 2230.209141] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 2230.222728] LNet: 1 local NIs in recovery (showing 1): 192.168.201.18@tcp [ 2230.749872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2230.810462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2230.861816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2230.927744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2230.991553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2231.058671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2231.140086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2231.203433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2231.242545] LNet: 88114:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2231.285133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2231.347672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2231.397069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2231.464127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2231.508678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2231.547472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2232.257826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_dropped -r 2 [ 2232.361807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp1 -m GET -e local_dropped -r 2 [ 2232.399679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_dropped -r 2 [ 2232.470237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -m GET -e local_dropped -r 2 [ 2232.518074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp -m GET -e local_dropped -r 2 [ 2232.580346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp1 -m GET -e local_dropped -r 2 [ 2232.650972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -m GET -e local_dropped -r 2 [ 2232.739661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -m GET -e local_dropped -r 2 [ 2232.812329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.18@tcp 192.168.201.118@tcp [ 2232.842661] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 2233.199591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2233.254224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2233.307126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2233.341950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2233.386946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2233.423445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2233.493975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2233.569421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2233.591516] LNet: 88234:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2233.625480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2233.658069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2233.698742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2233.731847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2233.790249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2233.836428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2234.412640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_aborted -r 2 [ 2234.492592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp1 -m GET -e local_aborted -r 2 [ 2234.523412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_aborted -r 2 [ 2234.550172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -m GET -e local_aborted -r 2 [ 2234.600986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp -m GET -e local_aborted -r 2 [ 2234.661574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp1 -m GET -e local_aborted -r 2 [ 2234.737244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -m GET -e local_aborted -r 2 [ 2234.798557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -m GET -e local_aborted -r 2 [ 2234.864075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.18@tcp 192.168.201.118@tcp [ 2234.905605] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 2235.216593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2235.267984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2235.311139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2235.344939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2235.369285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2235.397501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2235.433390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2235.472700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2235.497081] LNet: 88355:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2235.516718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2235.572629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2235.665043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2235.710045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2235.763684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2235.800332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2236.551752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_no_route -r 2 [ 2236.585673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp1 -m GET -e local_no_route -r 2 [ 2236.657436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_no_route -r 2 [ 2236.724855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -m GET -e local_no_route -r 2 [ 2236.791705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp -m GET -e local_no_route -r 2 [ 2236.877618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp1 -m GET -e local_no_route -r 2 [ 2236.950405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -m GET -e local_no_route -r 2 [ 2237.042726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -m GET -e local_no_route -r 2 [ 2237.083130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.18@tcp 192.168.201.118@tcp [ 2237.096514] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 2237.342990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2237.381364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2237.409311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2237.442266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2237.466942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2237.499816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2237.530950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2237.558937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2237.579620] LNet: 88475:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2237.597316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2237.633926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2237.663911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2237.695987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2237.736396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2237.774965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2238.556890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e local_timeout -r 2 [ 2238.608546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp1 -m GET -e local_timeout -r 2 [ 2238.658753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e local_timeout -r 2 [ 2238.702602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -m GET -e local_timeout -r 2 [ 2238.754846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp -m GET -e local_timeout -r 2 [ 2238.813607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp1 -m GET -e local_timeout -r 2 [ 2238.876248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -m GET -e local_timeout -r 2 [ 2238.979592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -m GET -e local_timeout -r 2 [ 2239.063758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.18@tcp 192.168.201.118@tcp [ 2239.493930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2239.560960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2239.617210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2239.677316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2239.712280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2239.784961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2239.841751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2239.929189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2239.991563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2240.074533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2240.137613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2240.234227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2240.280995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2240.344043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2241.230172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp -m GET -e remote_dropped -r 2 [ 2241.329228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.118@tcp1 -m GET -e remote_dropped -r 2 [ 2241.409946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp -m GET -e remote_dropped -r 2 [ 2241.500465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp -d 192.168.201.18@tcp1 -m GET -e remote_dropped -r 2 [ 2241.572682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp -m GET -e remote_dropped -r 2 [ 2241.663840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.118@tcp1 -m GET -e remote_dropped -r 2 [ 2241.711321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp -m GET -e remote_dropped -r 2 [ 2241.791361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.18@tcp1 -d 192.168.201.18@tcp1 -m GET -e remote_dropped -r 2 [ 2241.856552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.18@tcp 192.168.201.118@tcp [ 2241.898819] LNet: There was an unexpected network error while writing to 192.168.201.118: rc = -22 [ 2241.908504] LNet: Skipped 1 previous similar message [ 2241.915614] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.118@tcp [ 2242.257055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2242.303359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2242.367886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2242.448439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2242.520511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2242.587136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2242.643295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2242.725625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2242.751714] LNet: 88715:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2242.762864] LNet: 88715:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 2242.796512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2242.856991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2242.946941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2243.020593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2243.080808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2243.175021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2250.898397] LNet: 88791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2250.903771] LNetError: 88791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2251.936312] LNet: Removed LNI 192.168.201.18@tcp [ 2251.940796] LNetError: 88791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2251.958059] LNet: Removed LNI 192.168.201.18@tcp1 [ 2252.660711] Key type .llcrypt unregistered [ 2252.665402] Key type ._llcrypt unregistered [ 2276.482339] Key type ._llcrypt registered [ 2276.488787] Key type .llcrypt registered [ 2278.189315] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:53:44 (1785776024) [ 2278.473354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2278.486547] alg: No test for adler32 (adler32-zlib) [ 2279.611145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2279.788039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2279.865244] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2281.304499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2281.387831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.18@tcp [ 2281.448084] LNetError: 89787:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.18@tcp to recovery [ 2281.629727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.18@tcp [ 2281.778601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2293.049281] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:53:59 (1785776039) [ 2293.212862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2293.274204] LNet: 90371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.285603] LNetError: 90371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2293.308138] LNet: Removed LNI 192.168.201.18@tcp [ 2293.352181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.457441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2293.484922] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2294.408466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2294.435043] LNet: Added LNI 192.168.201.18@tcp1 [8/256/0/180] [ 2295.819859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.18@tcp [ 2295.862298] LNetError: 90644:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.18@tcp to recovery [ 2296.544526] LNet: 1 local NIs in recovery (showing 1): 192.168.201.18@tcp [ 2296.547914] LNet: There was an unexpected network error while writing to 192.168.201.18: rc = -22 [ 2297.142192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.18@tcp [ 2307.770020] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:54:13 (1785776053) [ 2309.815957] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg118-server [ 2312.343383] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:54:18 (1785776058) [ 2314.114389] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg118-server [ 2316.151189] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:54:22 (1785776062) [ 2318.819593] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg118-server [ 2321.329622] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:54:27 (1785776067) [ 2321.697856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2321.755115] LNet: 92274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2321.767845] LNetError: 92274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2321.781475] LNet: Removed LNI 192.168.201.18@tcp [ 2321.820468] LNet: Removed LNI 192.168.201.18@tcp1 [ 2321.863309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2321.988518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2322.024406] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2323.573554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2323.694466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2323.785646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2323.906620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2323.987257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2324.081088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2332.732217] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:54:38 (1785776078) [ 2332.915494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2332.982470] LNet: 92996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2332.992699] LNetError: 92996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2333.002180] LNetError: 92996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2333.038304] LNet: Removed LNI 192.168.201.18@tcp [ 2333.124623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.237538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2333.307585] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2343.160704] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:54:49 (1785776089) [ 2343.539917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.572682] LNet: 93706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2343.603577] LNetError: 93706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2343.649011] LNet: Removed LNI 192.168.201.18@tcp [ 2343.685840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.775372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2343.819576] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2351.787127] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:54:58 (1785776098) [ 2353.565261] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2355.863823] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:55:02 (1785776102) [ 2356.409499] LNet: 94741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2356.413750] LNetError: 94741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2356.430552] LNet: Removed LNI 192.168.201.18@tcp [ 2357.246747] Key type .llcrypt unregistered [ 2357.247813] Key type ._llcrypt unregistered [ 2366.046602] Key type ._llcrypt registered [ 2366.047758] Key type .llcrypt registered [ 2366.311922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2366.340120] alg: No test for adler32 (adler32-zlib) [ 2367.263547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2367.297795] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2375.926291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.118@tcp [ 2383.086547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.118@tcp [ 2383.443776] LNet: 95159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2383.449832] LNetError: 95159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2383.461827] LNet: Removed LNI 192.168.201.18@tcp [ 2383.823550] Key type .llcrypt unregistered [ 2383.824819] Key type ._llcrypt unregistered [ 2397.594556] Key type ._llcrypt registered [ 2397.596583] Key type .llcrypt registered [ 2399.054131] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2400.690041] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2402.506674] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:55:49 (1785776149) [ 2404.363578] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2406.869388] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:55:53 (1785776153) [ 2409.045199] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg118-server [ 2411.401568] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:55:57 (1785776157) [ 2413.258779] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg118-server [ 2415.595835] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:56:01 (1785776161) [ 2417.341052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2417.386416] alg: No test for adler32 (adler32-zlib) [ 2418.641860] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 2419.033612] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2420.872141] Key type lgssc registered [ 2423.096727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2448.858678] Key type lgssc unregistered [ 2449.226478] LNet: 98337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2449.234162] LNetError: 98337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2449.255209] LNet: Removed LNI 192.168.201.18@tcp [ 2450.367568] Key type .llcrypt unregistered [ 2450.372075] Key type ._llcrypt unregistered [ 2471.392324] Key type ._llcrypt registered [ 2471.396225] Key type .llcrypt registered [ 2472.823216] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:56:59 (1785776219) [ 2473.482846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2473.509871] alg: No test for adler32 (adler32-zlib) [ 2474.471443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2474.550635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2474.591974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2474.660983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2474.739443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2482.637859] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:57:08 (1785776228) [ 2482.729593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2482.787350] LNet: 99748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2482.818275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2482.902146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2482.920076] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2483.652818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2483.672730] LNet: 99897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2483.676647] LNetError: 99897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2483.699775] LNet: Removed LNI 192.168.201.18@tcp [ 2483.729910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2483.827645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2483.843337] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2484.673222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2484.722236] LNet: 99925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2484.736827] LNetError: 99925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2484.756418] LNet: Removed LNI 192.168.201.18@tcp [ 2484.790342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2484.910568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2484.971541] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2494.790438] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:57:21 (1785776241) [ 2495.066086] LNet: 100516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2495.081274] LNetError: 100516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2495.126368] LNet: Removed LNI 192.168.201.18@tcp [ 2495.760518] Key type .llcrypt unregistered [ 2495.762188] Key type ._llcrypt unregistered [ 2504.182779] Key type ._llcrypt registered [ 2504.185686] Key type .llcrypt registered [ 2504.465202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2504.493501] alg: No test for adler32 (adler32-zlib) [ 2505.439708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2505.519713] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2505.523203] LNet: *** cfs_fail_loc=e003, val=100*** [ 2506.406481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2506.443562] LNet: *** cfs_fail_loc=e003, val=100*** [ 2506.449483] LNet: Skipped 2 previous similar messages [ 2506.462552] Ping reply received with mapping: [ 2506.468546] found metadata reply NI[192.168.201.18@tcp] [ 2506.475938] 12 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2506.483250] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2506.485903] 5a 5a 5a 5a 5a 5a 5a 5a [ 2506.707792] LNet: 100915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2506.717191] LNetError: 100915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2507.752584] LNet: Removed LNI 192.168.201.18@tcp [ 2508.847442] Key type .llcrypt unregistered [ 2508.859260] Key type ._llcrypt unregistered [ 2525.615133] Key type ._llcrypt registered [ 2525.616482] Key type .llcrypt registered [ 2527.104186] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:57:53 (1785776273) [ 2527.584388] Key type .llcrypt unregistered [ 2527.587783] Key type ._llcrypt unregistered [ 2538.985887] Key type ._llcrypt registered [ 2538.989560] Key type .llcrypt registered [ 2539.241446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2539.277537] alg: No test for adler32 (adler32-zlib) [ 2540.685777] LNetError: 101962:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2541.127185] Key type .llcrypt unregistered [ 2541.137584] Key type ._llcrypt unregistered [ 2550.400965] Key type ._llcrypt registered [ 2550.411121] Key type .llcrypt registered [ 2552.852414] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:58:18 (1785776298) [ 2554.561561] Key type .llcrypt unregistered [ 2554.568090] Key type ._llcrypt unregistered [ 2565.577050] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2565.588129] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2568.480155] Key type ._llcrypt registered [ 2568.484235] Key type .llcrypt registered [ 2568.745472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2568.764493] alg: No test for adler32 (adler32-zlib) [ 2569.733760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2569.815266] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2580.391756] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:58:47 (1785776327) [ 2580.612452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2580.655197] LNet: 103744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2580.665412] LNetError: 103744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2580.688452] LNet: Removed LNI 10.1.2.3@tcp [ 2580.759361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2581.666680] Key type .llcrypt unregistered [ 2581.668842] Key type ._llcrypt unregistered [ 2590.827920] Key type ._llcrypt registered [ 2590.829338] Key type .llcrypt registered [ 2591.268972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2591.296940] alg: No test for adler32 (adler32-zlib) [ 2592.423558] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2592.572668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2593.029740] LNet: 104035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2593.049289] LNetError: 104035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2594.088793] LNet: Removed LNI 192.168.201.18@tcp [ 2595.390344] Key type .llcrypt unregistered [ 2595.391790] Key type ._llcrypt unregistered [ 2617.806788] Key type ._llcrypt registered [ 2617.809623] Key type .llcrypt registered [ 2620.712418] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:59:25 (1785776365) [ 2622.351139] Key type .llcrypt unregistered [ 2622.353151] Key type ._llcrypt unregistered [ 2635.517106] Key type ._llcrypt registered [ 2635.520804] Key type .llcrypt registered [ 2636.026615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2636.103452] alg: No test for adler32 (adler32-zlib) [ 2644.567448] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:59:50 (1785776390) [ 2644.828152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2644.965422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2645.052656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2645.105901] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2646.223903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2646.282492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2646.301652] LNet: Removed LNI 192.168.201.18@tcp [ 2646.515914] LNet: 105784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2646.780590] Key type .llcrypt unregistered [ 2646.783498] Key type ._llcrypt unregistered [ 2661.564653] Key type ._llcrypt registered [ 2661.567176] Key type .llcrypt registered [ 2663.293967] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 13:00:09 (1785776409) [ 2663.890320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2663.909505] alg: No test for adler32 (adler32-zlib) [ 2664.832560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2664.889797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2664.917717] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2665.855726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2665.943480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.18@tcp [ 2665.986505] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2665.990728] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2665.992835] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2665, 0, 0 [ 2665.998499] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2666.000672] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2666.003314] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2666.013645] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2665, 0, 0 [ 2666.022307] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2666.026818] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2666.029061] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2666.034923] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2665, 0, 2715 [ 2666.045365] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2666.049494] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2666.057873] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2666.062115] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2665, 0, 2715 [ 2666.076688] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2675.431233] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 13:00:21 (1785776421) [ 2676.375871] LNet: 107325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2676.402885] LNetError: 107325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2676.454135] LNet: Removed LNI 192.168.201.18@tcp [ 2677.233392] Key type .llcrypt unregistered [ 2677.234965] Key type ._llcrypt unregistered [ 2687.643356] Key type ._llcrypt registered [ 2687.650651] Key type .llcrypt registered [ 2688.281311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2688.326558] alg: No test for adler32 (adler32-zlib) [ 2689.437419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2689.495941] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2699.408443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.118@tcp [ 2712.660871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2712.742912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2712.800905] LNet: Added LNI 192.168.201.18@tcp1 [8/256/0/180] [ 2713.784951] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2713.922901] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2713.925857] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2713.929632] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2716.274820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2716.305692] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2727.641303] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 13:01:14 (1785776474) [ 2727.711793] LNet: 108837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2727.715318] LNetError: 108837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2727.720793] LNet: Removed LNI 192.168.201.18@tcp [ 2728.737324] LNet: Removed LNI 192.168.201.18@tcp1 [ 2728.739684] LNetError: 108837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2728.749840] LNetError: 108837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2728.764561] LNet: Removed LNI 10.1.2.3@tcp99 [ 2729.989477] Key type .llcrypt unregistered [ 2729.992285] Key type ._llcrypt unregistered [ 2741.653615] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2741.671395] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2743.442289] Key type ._llcrypt registered [ 2743.444793] Key type .llcrypt registered [ 2743.875178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2743.924992] alg: No test for adler32 (adler32-zlib) [ 2745.059375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2745.222345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2745.289062] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2746.895761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2746.935793] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2751.885588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2751.970224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2752.036863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2752.071390] LNetError: 109585:0:(peer.c:2277:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2752.113334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2752.184990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2753.005394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2753.153730] LNet: 109605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2753.163145] LNetError: 109605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2753.171134] LNet: Removed LNI 10.1.2.3@tcp [ 2754.214822] LNet: Removed LNI 192.168.201.18@tcp [ 2755.480429] Key type .llcrypt unregistered [ 2755.485317] Key type ._llcrypt unregistered [ 2773.770390] Key type ._llcrypt registered [ 2773.772560] Key type .llcrypt registered [ 2775.983138] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 13:02:01 (1785776521) [ 2777.045664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2777.092609] alg: No test for adler32 (adler32-zlib) [ 2778.153665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2778.270969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2778.316656] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2779.445938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2779.580513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg118-client@tcp [ 2787.516679] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 13:02:13 (1785776533) [ 2787.664130] LNet: 111133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2787.678614] LNetError: 111133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2788.722635] LNet: Removed LNI 192.168.201.18@tcp [ 2790.135398] Key type .llcrypt unregistered [ 2790.145049] Key type ._llcrypt unregistered [ 2801.634858] Key type ._llcrypt registered [ 2801.637576] Key type .llcrypt registered [ 2801.823445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2801.864559] alg: No test for adler32 (adler32-zlib) [ 2802.880841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2802.964272] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2803.030473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2813.373951] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 13:02:39 (1785776559) [ 2815.570469] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2818.668356] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 13:02:44 (1785776564) [ 2820.519463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2822.334705] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 13:02:48 (1785776568) [ 2822.687378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2822.721351] LNet: 112561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2822.729563] LNetError: 112561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2822.749595] LNet: Removed LNI 192.168.201.18@tcp [ 2822.769834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2822.855556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2822.905579] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2824.291629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2824.418799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2832.898342] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 13:02:59 (1785776579) [ 2833.269433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2833.296803] LNet: 113271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2833.302427] LNetError: 113271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2833.323370] LNet: Removed LNI 192.168.201.18@tcp [ 2833.364379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2833.436835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2833.474649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2834.170483] Key type .llcrypt unregistered [ 2834.175245] Key type ._llcrypt unregistered [ 2845.930881] Key type ._llcrypt registered [ 2845.934448] Key type .llcrypt registered [ 2847.779216] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 13:03:13 (1785776593) [ 2848.526658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2848.566829] alg: No test for adler32 (adler32-zlib) [ 2849.586266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2849.699436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2849.748190] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2850.903662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2851.004656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2859.335267] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 13:03:26 (1785776606) [ 2859.432795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2859.481658] LNet: 114820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2859.495857] LNetError: 114820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2859.560076] LNet: Removed LNI 192.168.201.18@tcp [ 2859.607076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2859.717459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2859.768804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2866.117098] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 13:03:32 (1785776612) [ 2866.455668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2866.479892] LNet: 115407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2866.528789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2866.657366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2867.320731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2867.416696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2867.528556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2868.230566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2868.264610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2868.362587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2868.411562] LNet: 115440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2868.465531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2868.568953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2869.267670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2869.355301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2869.496723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2870.067608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2870.108687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2878.499251] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 13:03:44 (1785776624) [ 2878.841285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2878.889151] LNet: 116040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2878.946632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2879.016844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2879.093155] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2880.269939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2880.300630] LNet: Added LNI 192.168.201.18@tcp2 [8/256/0/180] [ 2881.563663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2881.591122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2881.740543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2881.753791] LNetError: 116320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2882.784784] LNet: Removed LNI 192.168.201.18@tcp [ 2882.791219] LNetError: 116320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2882.818632] LNet: Removed LNI 192.168.201.18@tcp2 [ 2882.840723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2882.911140] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2882.915869] LNet: Added LNI 192.168.201.18@tcp2 [8/256/0/180] [ 2882.920169] LNet: 116326:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2883.542816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2884.031473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2884.072313] LNet: 116363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2884.082805] LNet: 116363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2885.153018] LNet: Removed LNI 192.168.201.18@tcp [ 2885.162631] LNetError: 116363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2885.188261] LNetError: 116363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2885.220097] LNet: Removed LNI 192.168.201.18@tcp2 [ 2885.286736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2885.408168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2885.444916] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2886.478566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2886.505628] LNet: Added LNI 192.168.201.18@tcp2 [8/256/0/180] [ 2887.730612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2887.789708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2887.889474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2888.291383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2889.376223] LNet: Removed LNI 192.168.201.18@tcp [ 2889.378863] LNetError: 116648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2889.396097] LNetError: 116648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2889.423812] LNet: Removed LNI 192.168.201.18@tcp2 [ 2889.464057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2889.561535] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2889.581578] LNet: Added LNI 192.168.201.18@tcp2 [8/256/0/180] [ 2889.589611] LNet: 116654:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2890.576703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2890.903191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2890.935806] LNet: Removed LNI 192.168.201.18@tcp [ 2892.006815] LNet: Removed LNI 192.168.201.18@tcp2 [ 2892.060322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2892.179734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2892.216101] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2893.491497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2893.534291] LNet: Added LNI 192.168.201.18@tcp2 [8/256/0/180] [ 2894.924420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2895.042512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2895.500856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2895.577068] LNet: 116971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2895.599583] LNet: 116971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2896.672213] LNet: Removed LNI 192.168.201.18@tcp [ 2896.694538] LNet: Removed LNI 192.168.201.18@tcp2 [ 2896.753517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2896.873486] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2896.887700] LNet: Added LNI 192.168.201.18@tcp2 [8/256/0/180] [ 2896.893761] LNet: 116977:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2897.966360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2898.303335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2898.346810] LNetError: 117015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2898.358800] LNetError: 117015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2898.369701] LNet: Removed LNI 192.168.201.18@tcp [ 2898.405218] LNet: Removed LNI 192.168.201.18@tcp2 [ 2898.433489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2898.513870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2898.561356] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2899.728236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2899.768199] LNet: Added LNI 192.168.201.18@tcp2 [8/256/0/180] [ 2900.805533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2900.953960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2901.055199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2901.415797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2902.432794] LNet: Removed LNI 192.168.201.18@tcp [ 2903.527267] LNet: Removed LNI 192.168.201.18@tcp2 [ 2903.632477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2903.733354] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2903.740959] LNet: Added LNI 192.168.201.18@tcp2 [8/256/0/180] [ 2903.753387] LNet: 117305:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2904.630471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2904.950447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2904.988440] LNet: Removed LNI 192.168.201.18@tcp [ 2906.024283] LNet: Removed LNI 192.168.201.18@tcp2 [ 2906.100515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2906.232689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2906.266375] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2906.992206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2907.021201] LNet: Added LNI 192.168.201.18@tcp2 [8/256/0/180] [ 2908.617542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2908.684347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2908.800326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2909.334738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2910.432214] LNet: Removed LNI 192.168.201.18@tcp [ 2910.446935] LNet: Removed LNI 192.168.201.18@tcp2 [ 2910.509944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2910.609638] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2910.620450] LNet: Added LNI 192.168.201.18@tcp2 [8/256/0/180] [ 2910.631882] LNet: 117645:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2911.487247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2911.821826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2911.860371] LNet: 117697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2911.867355] LNet: 117697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2911.887840] LNet: Removed LNI 192.168.201.18@tcp [ 2911.907824] LNet: Removed LNI 192.168.201.18@tcp2 [ 2911.974233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2912.044503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2912.079970] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2913.292563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2913.325126] LNet: Added LNI 192.168.201.18@tcp2 [8/256/0/180] [ 2914.629498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2914.747046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2914.813594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2914.937071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2915.512752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2915.556709] LNetError: 117999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2915.566692] LNetError: 117999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2915.580138] LNet: Removed LNI 192.168.201.18@tcp [ 2916.665948] LNet: Removed LNI 192.168.201.18@tcp2 [ 2916.719292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2916.816022] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2916.827599] LNet: Added LNI 192.168.201.18@tcp2 [8/256/0/180] [ 2917.497829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2925.455809] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2927.898874] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 13:04:33 (1785776673) [ 2928.012490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2928.056120] LNet: Removed LNI 192.168.201.18@tcp [ 2929.128872] LNet: Removed LNI 192.168.201.18@tcp2 [ 2929.199745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2929.611518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2929.625735] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2930.362837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2930.395864] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2931.855883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2931.893920] LNet: Removed LNI 10.1.2.3@tcp [ 2931.915154] LNet: Removed LNI 192.168.201.18@tcp [ 2931.956624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2932.042132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2932.093117] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2932.099405] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2932.784823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2932.833596] LNet: Removed LNI 192.168.201.18@tcp [ 2932.861209] LNet: Removed LNI 10.1.2.3@tcp [ 2940.943546] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 13:04:47 (1785776687) [ 2941.195515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2941.252617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2941.315177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2941.334195] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2942.055140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2942.176700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2942.259743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2942.306990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2942.351271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2942.405871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2942.464160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2942.513199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2942.566486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2942.628294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2942.668869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.18@tcp [ 2942.870282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2950.945758] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 13:04:57 (1785776697) [ 2951.076236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2951.127589] LNet: 120374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2951.138105] LNet: 120374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2951.171992] LNetError: 120374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2951.179864] LNetError: 120374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2952.232569] LNet: Removed LNI 192.168.201.18@tcp [ 2952.313509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2952.494254] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2952.645605] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2952.654196] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2952.666508] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2955.194201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2955.240735] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 2956.474765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2956.511031] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2959.639435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.18@tcp [ 2959.705932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2959.776902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.18@tcp -m PUT -e local_timeout -r 1 [ 2959.895698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2959.982302] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2959.984308] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2960.348230] LNetError: 120442:0:(lib-move.c:4801:lnet_parse()) 192.168.201.18@tcp, src 192.168.201.18@tcp: Dropping PUT (error -108 looking up sender) [ 2961.383195] LNet: Removed LNI 10.1.2.3@tcp [ 2962.417928] LNet: Removed LNI 192.168.201.18@tcp [ 2963.312918] Key type .llcrypt unregistered [ 2963.314399] Key type ._llcrypt unregistered [ 2980.742179] Key type ._llcrypt registered [ 2980.744827] Key type .llcrypt registered [ 2981.865296] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 13:05:28 (1785776728) [ 2982.631329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2982.644782] alg: No test for adler32 (adler32-zlib) [ 2983.767100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2984.487242] LNet: 121756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2985.538536] Key type .llcrypt unregistered [ 2985.540941] Key type ._llcrypt unregistered [ 2995.481811] Key type ._llcrypt registered [ 2995.485662] Key type .llcrypt registered [ 2995.713284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2995.729868] alg: No test for adler32 (adler32-zlib) [ 2996.863938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2996.918574] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 3007.924553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.118@tcp [ 3008.237657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 3009.061208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.118@tcp [ 3009.259957] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.118@tcp [ 3009.602312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3009.685614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3009.731291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3009.770419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3009.864937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3009.965409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3010.021272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3010.085546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3010.108671] LNet: 122223:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3010.142806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3010.228363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3010.298567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3010.352455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3010.415223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3010.460477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3011.528617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3011.983943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3012.042282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3012.093677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3012.153843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3012.194805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3012.244257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3012.283251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3012.328320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3012.354050] LNet: 122340:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3012.384713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3012.426791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3012.479849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3012.525796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3012.583570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3012.633312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3013.587716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.118@tcp [ 3013.845223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3013.866380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3013.918657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3013.950284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3013.985534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3014.025261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3014.115893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3014.163727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3014.187633] LNet: 122458:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3014.219565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3014.295789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3014.355475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3014.391429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3014.445174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3014.466450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3015.156375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3015.339907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3015.386410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3015.420177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3015.442946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3015.467132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3015.495938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3015.521681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3015.539358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3015.571282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3015.608933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3015.647596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3015.682682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3015.709841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3015.734845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3016.121790] LNet: 122621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3016.127800] LNetError: 122621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3016.163046] LNet: Removed LNI 192.168.201.18@tcp [ 3016.864735] Key type .llcrypt unregistered [ 3016.868163] Key type ._llcrypt unregistered [ 3038.883890] Key type ._llcrypt registered [ 3038.885787] Key type .llcrypt registered [ 3040.465901] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 13:06:26 (1785776786) [ 3041.110989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3041.143933] alg: No test for adler32 (adler32-zlib) [ 3042.041156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3042.685422] LNet: 123510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3043.569566] Key type .llcrypt unregistered [ 3043.577547] Key type ._llcrypt unregistered [ 3054.251542] Key type ._llcrypt registered [ 3054.257286] Key type .llcrypt registered [ 3054.522806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3054.585761] alg: No test for adler32 (adler32-zlib) [ 3055.448289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3055.464822] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 3069.256596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.118@tcp [ 3083.129786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3083.168415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3083.237786] LNet: Added LNI 192.168.201.18@tcp1 [8/256/0/180] [ 3084.271647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 3085.014514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.118@tcp [ 3085.155057] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.118@tcp [ 3085.250563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3085.284449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3085.338369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3085.386665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3085.429108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3085.473772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3085.515351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3085.576942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3085.624214] LNet: 124165:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3085.651518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3085.695802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3085.762945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3085.831495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3085.898230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3085.952791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3086.685800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3086.967718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3086.995297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3087.038291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3087.075345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3087.112197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3087.141426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3087.174939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3087.220893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3087.244774] LNet: 124282:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3087.263313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3087.293209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3087.319539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3087.346188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3087.377997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3087.417930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3088.116243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.118@tcp [ 3088.563159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3088.603505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3088.643638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3088.668470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3088.701389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3088.750362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3088.800881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3088.846990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3088.881933] LNet: 124399:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3088.928487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3088.997768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3089.056218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3089.086375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3089.122693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3089.159983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3089.885732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3090.158781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3090.206181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3090.245315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3090.268238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3090.317424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3090.353671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3090.382230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3090.411862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3090.463578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3090.523584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3090.563466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3090.607591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3090.643117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3090.700667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3097.591658] LNet: 124598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3097.596738] LNetError: 124598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3097.601755] LNet: Removed LNI 192.168.201.18@tcp [ 3097.611828] LNet: Removed LNI 192.168.201.18@tcp1 [ 3098.262701] Key type .llcrypt unregistered [ 3098.265128] Key type ._llcrypt unregistered [ 3120.719929] Key type ._llcrypt registered [ 3120.725804] Key type .llcrypt registered [ 3122.337366] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 13:07:48 (1785776868) [ 3125.089185] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 3128.225306] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 13:07:53 (1785776873) [ 3140.359163] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 13:08:06 (1785776886) [ 3149.248929] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 13:08:15 (1785776895) [ 3158.953710] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 13:08:24 (1785776904) [ 3168.196553] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 13:08:34 (1785776914) [ 3178.819675] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 13:08:45 (1785776925) [ 3192.261775] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 13:08:56 (1785776936) [ 3204.194700] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 13:09:10 (1785776950) [ 3213.154904] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 13:09:19 (1785776959) [ 3223.085577] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 13:09:29 (1785776969) [ 3232.582771] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 13:09:39 (1785776979) [ 3242.070298] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 13:09:48 (1785776988) [ 3251.480646] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 13:09:58 (1785776998) [ 3259.846392] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 13:10:06 (1785777006) [ 3270.055381] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 13:10:15 (1785777015) [ 3281.547418] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 13:10:27 (1785777027) [ 3293.211903] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 13:10:39 (1785777039) [ 3301.581844] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 13:10:47 (1785777047) [ 3301.869384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3301.880948] alg: No test for adler32 (adler32-zlib) [ 3302.980651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3303.113416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3303.149740] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 3305.477292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3305.567457] LNet: 135612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3305.584045] LNetError: 135612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3305.611697] LNet: Removed LNI 192.168.201.18@tcp [ 3305.663785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3305.810556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 3305.879787] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 3319.890194] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 13:11:05 (1785777065) [ 3328.269702] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 13:11:14 (1785777074) [ 3338.214950] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 13:11:23 (1785777083) [ 3348.267026] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 13:11:34 (1785777094) [ 3355.743419] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 13:11:42 (1785777102) [ 3365.078649] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 13:11:51 (1785777111) [ 3378.408567] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 13:12:03 (1785777123) [ 3389.101751] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 13:12:15 (1785777135) [ 3397.474872] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 13:12:23 (1785777143) [ 3409.330163] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 13:12:35 (1785777155) [ 3417.637158] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 13:12:43 (1785777163) [ 3425.876077] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 13:12:52 (1785777172) [ 3437.583674] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 3188 sec ========== 13:13:02 (1785777182) [ 3438.361367] LNet: 143087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3438.395334] LNetError: 143087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3438.430763] LNet: Removed LNI 192.168.201.18@tcp [ 3440.692751] Key type .llcrypt unregistered [ 3440.698276] Key type ._llcrypt unregistered [ 3459.030665] Key type ._llcrypt registered [ 3459.047759] Key type .llcrypt registered [ 3460.083550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3460.115362] alg: No test for adler32 (adler32-zlib) [ 3461.616216] Lustre: Lustre: Build Version: 2.17.56_2_g1ec524e [ 3462.100976] LNet: Added LNI 192.168.201.18@tcp [8/256/0/180] [ 3463.816192] Key type lgssc registered [ 3466.291817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3559.097356] Lustre: Mounted lustre-client [ 3565.305927] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3574.491234] Lustre: Unmounted lustre-client [ 3610.485837] Key type lgssc unregistered [ 3610.818158] LNet: 146024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3610.823447] LNetError: 146024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3611.899938] LNet: Removed LNI 192.168.201.18@tcp [ 3612.565131] Key type .llcrypt unregistered [ 3612.567570] Key type ._llcrypt unregistered