[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 452070886 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002244] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.006739] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008007] pid_max: default: 32768 minimum: 301 [ 0.009120] LSM: Security Framework initializing [ 0.010034] Yama: becoming mindful. [ 0.011024] SELinux: Initializing. [ 0.012053] *** VALIDATE selinux *** [ 0.020126] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025211] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026124] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027086] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028082] *** VALIDATE tmpfs *** [ 0.030238] *** VALIDATE proc *** [ 0.031196] *** VALIDATE cgroup *** [ 0.032005] *** VALIDATE cgroup2 *** [ 0.033187] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034133] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036022] Spectre V2 : User space: Vulnerable [ 0.037004] Speculative Store Bypass: Vulnerable [ 0.040339] debug: unmapping init [mem 0xffffffffa3659000-0xffffffffa3660fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043767] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044027] ... version: 2 [ 0.045009] ... bit width: 48 [ 0.046008] ... generic registers: 4 [ 0.047009] ... value mask: 0000ffffffffffff [ 0.048016] ... max period: 00007fffffffffff [ 0.049007] ... fixed-purpose events: 3 [ 0.049922] ... event mask: 000000070000000f [ 0.051234] rcu: Hierarchical SRCU implementation. [ 0.053400] smp: Bringing up secondary CPUs ... [ 0.054476] x86: Booting SMP configuration: [ 0.055016] .... node #0, CPUs: #1 #2 #3 [ 0.058373] smp: Brought up 1 node, 4 CPUs [ 0.059714] smpboot: Max logical packages: 1 [ 0.060008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.228184] node 0 deferred pages initialised in 167ms [ 0.230169] devtmpfs: initialized [ 0.231170] x86/mm: Memory block size: 128MB [ 0.233233] gcov: version magic: 0x41383552 [ 0.235135] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.236047] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.237235] pinctrl core: initialized pinctrl subsystem [ 0.238126] [ 0.238435] ************************************************************* [ 0.239007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.240005] ** ** [ 0.241008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.242006] ** ** [ 0.243006] ** This means that this kernel is built to expose internal ** [ 0.244005] ** IOMMU data structures, which may compromise security on ** [ 0.245005] ** your system. ** [ 0.246005] ** ** [ 0.247006] ** If you see this message and you are not debugging the ** [ 0.248005] ** kernel, report this immediately to your vendor! ** [ 0.249005] ** ** [ 0.250005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.251006] ************************************************************* [ 0.253016] NET: Registered protocol family 16 [ 0.254433] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.255041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.256027] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.257507] cpuidle: using governor menu [ 0.259750] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.262478] PCI: Using configuration type 1 for base access [ 0.264131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.272091] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.273051] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.274193] cryptd: max_cpu_qlen set to 1000 [ 0.276290] ACPI: Added _OSI(Module Device) [ 0.277009] ACPI: Added _OSI(Processor Device) [ 0.278026] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.279011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.282728] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.288554] ACPI: Interpreter enabled [ 0.289058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.290037] ACPI: Using IOAPIC for interrupt routing [ 0.292180] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.294336] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.302719] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.304026] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.305008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.307064] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.311008] acpiphp: Slot [2] registered [ 0.312021] acpiphp: Slot [3] registered [ 0.313091] acpiphp: Slot [4] registered [ 0.314075] acpiphp: Slot [5] registered [ 0.316115] acpiphp: Slot [6] registered [ 0.317107] acpiphp: Slot [7] registered [ 0.318044] acpiphp: Slot [8] registered [ 0.319068] acpiphp: Slot [9] registered [ 0.319876] acpiphp: Slot [10] registered [ 0.321050] acpiphp: Slot [11] registered [ 0.321958] acpiphp: Slot [12] registered [ 0.323074] acpiphp: Slot [13] registered [ 0.324111] acpiphp: Slot [14] registered [ 0.326051] acpiphp: Slot [15] registered [ 0.326895] acpiphp: Slot [16] registered [ 0.327043] acpiphp: Slot [17] registered [ 0.327817] acpiphp: Slot [18] registered [ 0.329060] acpiphp: Slot [19] registered [ 0.330117] acpiphp: Slot [20] registered [ 0.331050] acpiphp: Slot [21] registered [ 0.331871] acpiphp: Slot [22] registered [ 0.333066] acpiphp: Slot [23] registered [ 0.333827] acpiphp: Slot [24] registered [ 0.335084] acpiphp: Slot [25] registered [ 0.336063] acpiphp: Slot [26] registered [ 0.336907] acpiphp: Slot [27] registered [ 0.337059] acpiphp: Slot [28] registered [ 0.337766] acpiphp: Slot [29] registered [ 0.338041] acpiphp: Slot [30] registered [ 0.339108] acpiphp: Slot [31] registered [ 0.340052] PCI host bridge to bus 0000:00 [ 0.341015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.343017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.344008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.346012] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.347007] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.349013] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.350000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.352383] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.354657] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.359936] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.362469] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.363013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.364011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.366009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.367424] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.368477] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.370019] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372516] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.373963] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.379665] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.386022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.389065] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.406020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.413027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.435031] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.443476] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.450029] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.459029] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.482020] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.491877] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.494357] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.497327] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.498202] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.500097] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.503037] iommu: Default domain type: Passthrough [ 0.504481] SCSI subsystem initialized [ 0.505092] ACPI: bus type USB registered [ 0.505985] usbcore: registered new interface driver usbfs [ 0.507029] usbcore: registered new interface driver hub [ 0.508039] usbcore: registered new device driver usb [ 0.509146] pps_core: LinuxPPS API ver. 1 registered [ 0.511005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.512035] PTP clock support registered [ 0.514086] EDAC MC: Ver: 3.0.0 [ 0.515365] PCI: Using ACPI for IRQ routing [ 0.517950] NetLabel: Initializing [ 0.519013] NetLabel: domain hash size = 128 [ 0.520009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.522078] NetLabel: unlabeled traffic allowed by default [ 0.524203] vgaarb: loaded [ 0.526105] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.527000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.532629] clocksource: Switched to clocksource kvm-clock [ 0.641368] VFS: Disk quotas dquot_6.6.0 [ 0.642903] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.645314] *** VALIDATE ramfs *** [ 0.646533] *** VALIDATE hugetlbfs *** [ 0.647980] pnp: PnP ACPI init [ 0.650643] pnp: PnP ACPI: found 6 devices [ 0.666977] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.668854] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.670137] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.671407] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.672844] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.674115] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.675591] NET: Registered protocol family 2 [ 0.677486] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.681975] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.684649] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.689494] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.692495] TCP: Hash tables configured (established 65536 bind 65536) [ 0.695027] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.697282] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.700262] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.703411] NET: Registered protocol family 1 [ 0.706398] RPC: Registered named UNIX socket transport module. [ 0.708159] RPC: Registered udp transport module. [ 0.709495] RPC: Registered tcp transport module. [ 0.711015] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.713251] NET: Registered protocol family 44 [ 0.715422] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.717903] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.720179] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.722133] PCI: CLS 0 bytes, default 64 [ 0.723766] Unpacking initramfs... [ 2.221940] debug: unmapping init [mem 0xffff96efbcc64000-0xffff96efbffcffff] [ 2.226533] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.228371] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.231287] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.790689] Initialise system trusted keyrings [ 2.791891] Key type blacklist registered [ 2.794300] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.802852] zbud: loaded [ 2.806450] *** VALIDATE nfs *** [ 2.807286] *** VALIDATE nfs4 *** [ 2.808276] pstore: using deflate compression [ 2.811749] Platform Keyring initialized [ 2.931305] NET: Registered protocol family 38 [ 2.932984] Key type asymmetric registered [ 2.934744] Asymmetric key parser 'x509' registered [ 2.936777] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.939193] io scheduler mq-deadline registered [ 2.940414] io scheduler kyber registered [ 2.941600] io scheduler bfq registered [ 2.943259] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.945486] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.948023] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.950538] ACPI: Power Button [PWRF] [ 3.060396] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.172329] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.275256] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.308375] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.340697] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.348855] Non-volatile memory driver v1.3 [ 3.350655] Linux agpgart interface v0.103 [ 3.383754] virtio_blk virtio1: [vda] 134040 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.386299] vda: detected capacity change from 0 to 68628480 [ 3.401645] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.404642] vdb: detected capacity change from 0 to 1073741824 [ 3.411130] libphy: Fixed MDIO Bus: probed [ 3.416502] usbcore: registered new interface driver usbserial_generic [ 3.419167] usbserial: USB Serial support registered for generic [ 3.422752] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.426467] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.428172] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.430679] mousedev: PS/2 mouse device common for all mice [ 3.434370] rtc_cmos 00:05: RTC can wake from S4 [ 3.438852] rtc_cmos 00:05: registered as rtc0 [ 3.440910] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.443462] intel_pstate: CPU model not supported [ 3.446083] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.451180] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.453989] hid: raw HID events driver (C) Jiri Kosina [ 3.476504] hpet1: lost 1 rtc interrupts [ 3.481868] usbcore: registered new interface driver usbhid [ 3.484443] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.485691] usbhid: USB HID core driver [ 3.489786] drop_monitor: Initializing network drop monitor service [ 3.491154] Initializing XFRM netlink socket [ 3.493792] NET: Registered protocol family 10 [ 3.497198] Segment Routing with IPv6 [ 3.498413] NET: Registered protocol family 17 [ 3.499884] mpls_gso: MPLS GSO support [ 3.505525] RAS: Correctable Errors collector initialized. [ 3.507185] AVX version of gcm_enc/dec engaged. [ 3.508626] AES CTR mode by8 optimization enabled [ 3.614573] sched_clock: Marking stable (3614553483, 0)->(4445044967, -830491484) [ 3.618110] registered taskstats version 1 [ 3.620442] Loading compiled-in X.509 certificates [ 3.622986] zswap: loaded using pool lzo/zbud [ 3.654580] Key type big_key registered [ 3.687507] Key type encrypted registered [ 3.688972] ima: No TPM chip found, activating TPM-bypass! [ 3.690758] ima: Allocated hash algorithm: sha1 [ 3.692458] ima: No architecture policies found [ 3.695576] evm: Initialising EVM extended attributes: [ 3.697639] evm: security.selinux [ 3.698944] evm: security.ima [ 3.700075] evm: security.capability [ 3.701342] evm: HMAC attrs: 0x1 [ 3.705436] rtc_cmos 00:05: setting system clock to 2025-12-29 18:54:53 UTC (1767034493) [ 3.713490] debug: unmapping init [mem 0xffffffffa4603000-0xffffffffa47fffff] [ 3.717979] debug: unmapping init [mem 0xffffffffa3382000-0xffffffffa3658fff] [ 3.727307] Write protecting the kernel read-only data: 28672k [ 3.732982] debug: unmapping init [mem 0xffffffffa1a03000-0xffffffffa1bfffff] [ 3.735800] debug: unmapping init [mem 0xffffffffa2314000-0xffffffffa23fffff] [ 3.773669] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.783115] systemd[1]: Detected virtualization kvm. [ 3.785083] systemd[1]: Detected architecture x86-64. [ 3.786595] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.828709] systemd[1]: No hostname configured. [ 3.830145] systemd[1]: Set hostname to . [ 3.831755] random: systemd: uninitialized urandom read (16 bytes read) [ 3.837361] systemd[1]: Initializing machine ID from random generator. [ 3.981510] random: systemd: uninitialized urandom read (16 bytes read) [ 3.983445] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.990087] random: systemd: uninitialized urandom read (16 bytes read) [ 3.992642] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.031688] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.925788] device-mapper: uevent: version 1.0.3 [ 4.927743] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 5.919349] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.966363] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.099497] scsi host0: ata_piix [ 6.149471] scsi host1: ata_piix [ 6.150944] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.157371] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 11.237748] random: crng init done [ 11.247724] random: 7 urandom warning(s) missed due to ratelimiting [ 11.391318] dracut-initqueue[573]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.192296] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.963104] printk: systemd: 25 output lines suppressed due to ratelimiting [ 15.360626] SELinux: Disabled at runtime. [ 15.452094] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.460605] systemd[1]: Detected virtualization kvm. [ 15.462109] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.267029] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.270797] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.282159] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.288410] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.291637] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.301202] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.323663] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 16.390451] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 17.127051] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.662285] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.716577] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.007881] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.043393] EDAC sbridge: Ver: 1.1.2 [ 19.449921] Key type dns_resolver registered [ 19.841746] NFS: Registering the id_resolver key type [ 19.842915] Key type id_resolver registered [ 19.843726] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg240-client login: [ 48.296854] libcfs: loading out-of-tree module taints kernel. [ 48.343413] Key type ._llcrypt registered [ 48.344291] Key type .llcrypt registered [ 48.681416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 48.686890] alg: No test for adler32 (adler32-zlib) [ 49.669646] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 49.938870] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 51.544104] Key type lgssc registered [ 52.075525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.918246] Lustre: Mounted lustre-client [ 99.316611] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 112.863287] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing check_logdir /tmp/testlogs/ [ 114.332152] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing yml_node [ 115.674953] Lustre: DEBUG MARKER: Client: 2.17.0.RC3 [ 116.481537] Lustre: DEBUG MARKER: MDS: 2.17.0.RC3 [ 117.448499] Lustre: DEBUG MARKER: OSS: 2.17.0.RC3 [ 118.092826] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Dec 29 13:56:47 EST 2025 [ 122.848429] Lustre: lustre-OST0000-osc-ffff96f018082000: disconnect after 24s idle [ 125.700877] hrtimer: interrupt took 2909692 ns [ 131.450444] LustreError: 7917:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff96f018082000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 131.487519] LustreError: 7917:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 131.527173] Lustre: Unmounted lustre-client [ 161.415899] Key type lgssc unregistered [ 162.027180] LNet: 8364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 162.057980] LNetError: 8364:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 162.094385] LNet: Removed LNI 192.168.202.40@tcp [ 163.258615] Key type .llcrypt unregistered [ 163.260292] Key type ._llcrypt unregistered [ 189.226638] Key type ._llcrypt registered [ 189.228237] Key type .llcrypt registered [ 189.368750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 189.381134] alg: No test for adler32 (adler32-zlib) [ 190.273668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 190.314265] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 190.337119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 193.052401] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 193.167173] LNet: 9249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 193.176975] LNetError: 9249:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 193.203967] LNet: Removed LNI 192.168.202.40@tcp [ 193.731497] Key type .llcrypt unregistered [ 193.733084] Key type ._llcrypt unregistered [ 197.280417] Key type ._llcrypt registered [ 197.282014] Key type .llcrypt registered [ 198.204367] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 13:58:06 (1767034686) [ 198.512171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 198.526862] alg: No test for adler32 (adler32-zlib) [ 199.373595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 199.435572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 199.454489] LNet: 9658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 199.468240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 199.502727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 199.513914] LNet: 9665:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 203.866819] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 13:58:12 (1767034692) [ 203.927800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 203.955348] LNet: 10239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 203.970426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 204.014797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 207.668802] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 13:58:16 (1767034696) [ 207.796167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 207.819838] LNet: 10826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 207.834038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 207.885904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 211.310774] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 13:58:20 (1767034700) [ 211.362767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 211.374560] LNet: 11406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 211.394321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 211.443315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 215.000911] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 13:58:23 (1767034703) [ 215.052739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 215.112101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 215.173621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 215.453746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 218.899148] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 13:58:28 (1767034708) [ 218.994550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 219.019205] LNet: 12582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 219.022450] LNet: 12582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 219.037610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 219.088198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 219.325343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 219.552520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 223.383947] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 13:58:32 (1767034712) [ 223.487023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.506625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.545668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 226.953550] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 13:58:35 (1767034715) [ 227.005726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 227.026574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 227.069414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 227.089939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 227.263318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 227.280914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 227.473957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 227.490863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 227.693702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 227.716654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 227.923257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 227.947241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 228.136707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 228.153824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 228.330715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 228.347303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 228.521991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 228.536552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 228.713429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 228.733027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 228.922173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 228.942094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 229.136569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 229.153804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 229.349313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 229.370268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 229.559502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 229.578059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 232.974442] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 13:58:41 (1767034721) [ 233.033586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.053755] LNet: 14439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 233.056490] LNet: 14439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 233.078468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.151684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 233.182740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 236.987111] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 13:58:45 (1767034725) [ 237.047389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 237.088276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 237.144805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 237.194785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 241.363334] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 13:58:50 (1767034730) [ 241.479658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 241.512512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 241.563960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 241.582233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 245.722836] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 13:58:54 (1767034734) [ 245.765352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 245.798487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 245.842677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 245.879842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 249.975599] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 13:58:58 (1767034738) [ 250.027692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 250.058664] LNet: 16780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 250.063518] LNet: 16780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 250.087628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 250.155683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 250.185538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 254.677596] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 13:59:03 (1767034743) [ 254.819891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 254.858997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 254.930269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 254.987576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 259.520503] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 13:59:08 (1767034748) [ 259.608562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 259.644313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 259.707910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 259.948139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 264.141655] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 13:59:13 (1767034753) [ 264.197219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 264.229599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 264.370240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 264.582560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 268.512497] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 13:59:17 (1767034757) [ 268.572414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.603258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.692836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 268.833201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 272.463875] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 13:59:21 (1767034761) [ 272.543635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 272.579331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.865688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 273.000520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 276.470441] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 13:59:25 (1767034765) [ 276.520119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 276.550195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.592653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 276.620489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 280.099928] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 13:59:29 (1767034769) [ 280.189714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 280.220670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 280.262130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 280.287120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 283.752326] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 13:59:32 (1767034772) [ 283.806440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 283.819188] LNet: 21636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 283.823028] LNet: 21636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 283.841220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 283.884378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 287.234884] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 13:59:36 (1767034776) [ 287.312160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 287.336671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 287.380902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 290.935811] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 13:59:39 (1767034779) [ 291.029304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 291.047892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 291.085921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 291.277176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 294.271397] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 13:59:43 (1767034783) [ 294.326381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 294.364792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.404407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 294.421732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 297.373424] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 13:59:46 (1767034786) [ 297.438308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.468297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.506652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 297.525492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 300.488398] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 13:59:49 (1767034789) [ 300.533221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.554842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.604504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 300.622873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 303.342940] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 13:59:52 (1767034792) [ 303.379506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.399844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.445174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 303.463353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 306.139804] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 13:59:55 (1767034795) [ 306.173404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.193872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.223986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 306.252629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 306.262252] LNetError: 25750:0:(peer.c:2225:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 306.281449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 306.298470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 308.897532] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 13:59:58 (1767034798) [ 308.967928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.988975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.045365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.27GTBhg5Iz [ 309.082845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.27GTBhg5Iz [ 309.115448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.27GTBhg5Iz [ 312.618598] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 14:00:01 (1767034801) [ 312.731725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 312.776423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 312.810977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 312.841604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 312.869376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 316.865662] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 14:00:05 (1767034805) [ 316.915831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 316.945968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.984290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 318.283143] LNet: Message forwarding will be enabled in 2s [ 321.379631] LNet: Message forwarding enabled [ 321.543361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 321.558992] LNet: Message forwarding will stop in 2s to 6s [ 325.774985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 326.195711] LNet: Message forwarding disabled [ 326.900746] LNet: Message forwarding will be enabled in 2s [ 329.249092] LNet: Message forwarding enabled [ 331.278384] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 14:00:20 (1767034820) [ 331.338141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.289761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.346832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 336.402670] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 14:00:25 (1767034825) [ 336.494431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.522159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.562657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 336.587131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 336.607738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 336.627674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 336.655528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 336.673019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 336.691642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 336.714368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 336.736569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 336.760076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 336.792214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 336.809788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 336.832746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 336.862698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 336.890468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 336.922138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 336.939767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 336.975456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 337.008610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 337.022768] LNetError: 28818:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 340.710589] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 14:00:29 (1767034829) [ 340.777393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.803840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.850849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 344.433344] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 14:00:33 (1767034833) [ 345.469521] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg240-server [ 346.580156] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 14:00:35 (1767034835) [ 346.659388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.683251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.722419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 346.742141] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 347.767806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[41-45]@tcp [ 347.780132] LNet: 30536:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 347.820833] LNetError: 30323:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.43@tcp is being used as a gateway but routing feature is not turned on [ 347.837157] LNetError: 30323:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.43@tcp has gone from up to down [ 348.842083] LNetError: 30323:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.43@tcp is being used as a gateway but routing feature is not turned on [ 348.848891] LNetError: 30323:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 349.868986] LNetError: 30323:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.44@tcp is being used as a gateway but routing feature is not turned on [ 349.875970] LNetError: 30323:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 350.881485] LNetError: 30331:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.42@tcp has gone from up to down [ 350.896361] LNetError: 30331:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 351.366684] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 14:00:40 (1767034840) [ 351.915622] LNetError: 30323:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.44@tcp is being used as a gateway but routing feature is not turned on [ 351.926246] LNetError: 30323:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 352.409636] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg240-server [ 353.517573] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 14:00:42 (1767034842) [ 353.622943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.952168] LNetError: 30322:0:(router.c:1267:lnet_check_routers()) Failed to discover router 192.168.202.43@tcp [ 354.656601] LNet: 31470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 354.662580] LNet: 31470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 354.668969] LNetError: 31470:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 354.680580] LNet: Removed LNI 192.168.202.40@tcp [ 354.695716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.740381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 354.754101] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 355.195620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[41-46/2]@tcp [ 355.210275] LNet: 31587:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 355.213860] LNet: 31587:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 355.229376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[41-46/2]@tcp [ 355.235301] LNetError: 31474:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.43@tcp has gone from up to down [ 355.253658] LNetError: 31474:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 359.068429] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 14:00:47 (1767034847) [ 359.175292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.200503] LNetError: 32177:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 359.211879] LNet: Removed LNI 192.168.202.40@tcp [ 359.230242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.321864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 359.358306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.381517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.435323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 359.479261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.514725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.547990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 359.609752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.629545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.667960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 359.719967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.744459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.785081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 359.825613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.850510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.888490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 359.933470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.961202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.991887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 360.065611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.095918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.142793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 360.195716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.228288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.265391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 360.313204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.339277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.388605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 360.440440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.459429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.491620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 360.520793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.539882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.579342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 364.060210] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 14:00:53 (1767034853) [ 365.170530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg240-server [ 366.157893] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 14:00:55 (1767034855) [ 367.332637] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg240-server [ 368.401526] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 14:00:57 (1767034857) [ 368.901080] Key type .llcrypt unregistered [ 368.902717] Key type ._llcrypt unregistered [ 372.385589] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 372.495279] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 372.499713] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 372.502139] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 374.850441] Key type ._llcrypt registered [ 374.852138] Key type .llcrypt registered [ 374.964474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 374.974905] alg: No test for adler32 (adler32-zlib) [ 375.822544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.859328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 375.886634] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 376.332901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 376.351634] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 377.530494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 377.543516] LNet: Removed LNI 10.1.2.3@tcp [ 377.604321] LNet: 34303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.608543] LNetError: 34303:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 377.621665] LNet: Removed LNI 192.168.202.40@tcp [ 377.987489] Key type .llcrypt unregistered [ 377.988950] Key type ._llcrypt unregistered [ 383.588158] Key type ._llcrypt registered [ 383.589840] Key type .llcrypt registered [ 384.354935] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 14:01:13 (1767034873) [ 384.670597] Key type .llcrypt unregistered [ 384.672232] Key type ._llcrypt unregistered [ 387.834132] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 387.958916] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 387.961677] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 387.965899] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 390.291415] Key type ._llcrypt registered [ 390.292894] Key type .llcrypt registered [ 390.395870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 390.410292] alg: No test for adler32 (adler32-zlib) [ 391.261844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.292669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 391.309578] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 391.783677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 391.801978] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 393.559211] LNet: 35786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.563904] LNetError: 35786:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 393.567359] LNet: Removed LNI 192.168.202.40@tcp [ 393.582615] LNet: Removed LNI 10.1.2.3@tcp [ 394.005683] Key type .llcrypt unregistered [ 394.010304] Key type ._llcrypt unregistered [ 399.971972] Key type ._llcrypt registered [ 399.973955] Key type .llcrypt registered [ 400.762733] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 14:01:29 (1767034889) [ 401.013900] Key type .llcrypt unregistered [ 401.015990] Key type ._llcrypt unregistered [ 403.979558] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 404.026721] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 404.028496] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 404.030881] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 406.494077] Key type ._llcrypt registered [ 406.495550] Key type .llcrypt registered [ 406.634342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 406.646064] alg: No test for adler32 (adler32-zlib) [ 407.521419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.561431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 407.578783] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 408.939680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 408.957641] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 410.343736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 410.367835] LNet: Removed LNI 10.1.2.3@tcp [ 410.380284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 410.408779] LNet: Removed LNI 10.1.2.31@tcp [ 410.419092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 410.444045] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 411.075578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 411.096400] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 411.776277] LNet: 37699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.779967] LNetError: 37699:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 411.784224] LNet: Removed LNI 10.1.2.3@tcp [ 411.794646] LNet: Removed LNI 10.1.2.31@tcp [ 412.231791] Key type .llcrypt unregistered [ 412.233427] Key type ._llcrypt unregistered [ 418.917473] Key type ._llcrypt registered [ 418.918941] Key type .llcrypt registered [ 419.722775] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 14:01:48 (1767034908) [ 419.968900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 419.983623] alg: No test for adler32 (adler32-zlib) [ 420.843209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.875159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 420.895724] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 421.447914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.470350] LNet: 38646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.473494] LNetError: 38646:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 421.483879] LNet: Removed LNI 192.168.202.40@tcp [ 421.496588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.535975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.40@tcp [ 421.555296] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 421.994650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.40@tcp [ 422.022345] LNet: Removed LNI 192.168.202.40@tcp [ 422.069665] LNet: 38781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.423498] Key type .llcrypt unregistered [ 422.425270] Key type ._llcrypt unregistered [ 428.734952] Key type ._llcrypt registered [ 428.736560] Key type .llcrypt registered [ 429.535284] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 14:01:58 (1767034918) [ 430.796366] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg240-server [ 432.028286] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 14:02:00 (1767034920) [ 432.323429] Key type .llcrypt unregistered [ 432.325059] Key type ._llcrypt unregistered [ 436.095715] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 436.224637] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 436.232046] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 436.237109] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 438.590784] Key type ._llcrypt registered [ 438.592595] Key type .llcrypt registered [ 438.723728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 438.733841] alg: No test for adler32 (adler32-zlib) [ 439.590784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.628635] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 440.146745] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 441.615988] LNet: 40602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 441.625693] LNetError: 40602:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 441.629406] LNet: Removed LNI 192.168.202.40@tcp1 [ 441.643157] LNet: Removed LNI 10.1.2.3@tcp2 [ 442.015684] Key type .llcrypt unregistered [ 442.018822] Key type ._llcrypt unregistered [ 448.141793] Key type ._llcrypt registered [ 448.143441] Key type .llcrypt registered [ 448.908124] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 14:02:17 (1767034937) [ 449.094238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 449.102996] alg: No test for adler32 (adler32-zlib) [ 449.937771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.971875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 449.986758] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 450.500703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.514265] LNet: 41567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 450.518667] LNetError: 41567:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 450.534768] LNet: Removed LNI 192.168.202.40@tcp [ 450.546480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.581053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 450.593502] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 454.053517] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 14:02:23 (1767034943) [ 454.106969] LNet: 42152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 454.110803] LNetError: 42152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 454.121501] LNet: Removed LNI 192.168.202.40@tcp [ 454.410399] Key type .llcrypt unregistered [ 454.411784] Key type ._llcrypt unregistered [ 457.243107] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 457.278429] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 457.281020] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 457.283563] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 459.624848] Key type ._llcrypt registered [ 459.625948] Key type .llcrypt registered [ 459.724993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 459.736164] alg: No test for adler32 (adler32-zlib) [ 460.586156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.623283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 460.639742] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 461.067805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 461.080444] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 462.388964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.407191] LNet: 42895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 462.413222] LNetError: 42895:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 462.416716] LNet: Removed LNI 192.168.202.40@tcp [ 462.433093] LNet: Removed LNI 10.1.2.3@tcp [ 462.450910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.499952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 462.523613] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 462.528073] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 462.836364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.853356] LNet: Removed LNI 192.168.202.40@tcp [ 462.867404] LNet: Removed LNI 10.1.2.3@tcp [ 462.883538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.934955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 462.962233] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 462.964691] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 463.226586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 463.239608] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 463.820474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.835552] LNet: 43084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 463.838934] LNet: 43084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 463.842903] LNetError: 43084:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 463.846197] LNetError: 43084:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 463.849745] LNet: Removed LNI 192.168.202.40@tcp [ 463.851282] LNet: Removed LNI 10.1.2.3@tcp [ 463.859524] LNet: Removed LNI 192.168.202.40@tcp2 [ 463.880438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.926672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 463.948884] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 463.952455] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 463.958793] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 468.380290] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 14:02:37 (1767034957) [ 468.451518] LNet: 43675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.454879] LNetError: 43675:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 468.458130] LNetError: 43675:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 468.469410] LNet: Removed LNI 192.168.202.40@tcp [ 468.471841] LNet: Removed LNI 10.1.2.3@tcp [ 468.489486] LNet: Removed LNI 192.168.202.40@tcp2 [ 468.960257] Key type .llcrypt unregistered [ 468.961874] Key type ._llcrypt unregistered [ 472.359406] Key type ._llcrypt registered [ 472.361120] Key type .llcrypt registered [ 472.581770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 472.594710] alg: No test for adler32 (adler32-zlib) [ 473.432137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.464912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 473.486754] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 474.010227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.026563] LNet: 44072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.031176] LNetError: 44072:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 474.047673] LNet: Removed LNI 192.168.202.40@tcp [ 474.060846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.097869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 474.112882] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 474.356418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.384141] LNet: Removed LNI 192.168.202.40@tcp [ 474.399303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.437500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 474.455385] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 474.714343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.728936] LNet: 44132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.732370] LNet: 44132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 474.740144] LNetError: 44132:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 474.743455] LNetError: 44132:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 474.758747] LNet: Removed LNI 192.168.202.40@tcp [ 474.775102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.814915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 474.834921] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 475.090680] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 475.124278] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 475.126399] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 475.128834] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 477.379730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.393241] LNet: 44210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 477.398125] LNetError: 44210:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 477.407535] LNet: Removed LNI 192.168.202.40@tcp [ 477.420420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.455765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 477.470602] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 480.844109] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 14:02:49 (1767034969) [ 480.913526] LNet: 44803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 480.916427] LNetError: 44803:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 480.925700] LNet: Removed LNI 192.168.202.40@tcp [ 481.224979] Key type .llcrypt unregistered [ 481.227141] Key type ._llcrypt unregistered [ 483.821841] Key type ._llcrypt registered [ 483.823376] Key type .llcrypt registered [ 483.916770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 483.926169] alg: No test for adler32 (adler32-zlib) [ 484.757618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.789643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 484.801642] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 485.236381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.245772] LNet: 45200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 485.249368] LNetError: 45200:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 485.258081] LNet: Removed LNI 192.168.202.40@tcp [ 485.270055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.303469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 485.321452] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 488.442851] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 14:02:57 (1767034977) [ 488.510924] LNet: 45788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 488.516690] LNetError: 45788:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 488.527861] LNet: Removed LNI 192.168.202.40@tcp [ 488.814562] Key type .llcrypt unregistered [ 488.816058] Key type ._llcrypt unregistered [ 491.481590] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 491.539568] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 491.543534] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 491.546373] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 493.807334] Key type ._llcrypt registered [ 493.808947] Key type .llcrypt registered [ 493.907021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 493.919226] alg: No test for adler32 (adler32-zlib) [ 494.768547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.804061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 494.818928] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 495.270945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.281836] LNet: 46237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 495.285235] LNetError: 46237:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 495.298456] LNet: Removed LNI 192.168.202.40@tcp [ 495.308781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.344219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 495.359923] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 498.931124] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 14:03:07 (1767034987) [ 498.992887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.006478] LNet: 46829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 499.010740] LNetError: 46829:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 499.019512] LNet: Removed LNI 192.168.202.40@tcp [ 499.036143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.104830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 502.609962] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 14:03:11 (1767034991) [ 502.665751] LNet: 47407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 503.033235] Key type .llcrypt unregistered [ 503.034483] Key type ._llcrypt unregistered [ 505.828475] Key type ._llcrypt registered [ 505.829935] Key type .llcrypt registered [ 505.936804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 505.953865] alg: No test for adler32 (adler32-zlib) [ 506.779170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.807986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 506.821926] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 507.251391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.264975] LNet: 47808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 507.270881] LNetError: 47808:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 507.283590] LNet: Removed LNI 192.168.202.40@tcp [ 507.296228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.340411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 507.355360] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 510.698529] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 14:03:19 (1767034999) [ 510.778853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.791109] LNet: 48400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 510.793824] LNetError: 48400:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 510.806162] LNet: Removed LNI 192.168.202.40@tcp [ 510.816781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.858466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 513.673231] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 14:03:22 (1767035002) [ 514.365354] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 515.147378] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 14:03:24 (1767035004) [ 515.922638] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 516.743509] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 14:03:25 (1767035005) [ 517.500307] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 518.229772] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 14:03:27 (1767035007) [ 518.920679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 519.690760] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 14:03:28 (1767035008) [ 520.439219] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 521.217815] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 14:03:30 (1767035010) [ 521.306480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.317832] LNet: 50451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.331170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.362863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 521.377862] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 521.840187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.40 [ 521.851321] LNet: 50584:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 521.856637] LNet: Removed LNI 192.168.202.40@tcp [ 524.797881] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 14:03:33 (1767035013) [ 524.866779] LNet: 51148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 524.869381] LNetError: 51148:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 524.878146] LNet: Removed LNI 192.168.202.40@tcp [ 525.188911] Key type .llcrypt unregistered [ 525.190455] Key type ._llcrypt unregistered [ 527.835833] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 527.950370] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 527.953560] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 527.956801] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 530.279870] Key type ._llcrypt registered [ 530.281256] Key type .llcrypt registered [ 530.355645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 530.363879] alg: No test for adler32 (adler32-zlib) [ 531.403029] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 534.687562] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 14:03:43 (1767035023) [ 534.733283] LNet: 52047:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 534.735992] LNetError: 52047:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 534.745554] LNet: Removed LNI 10.1.2.3@tcp [ 535.108145] Key type .llcrypt unregistered [ 535.109799] Key type ._llcrypt unregistered [ 537.989164] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 538.123883] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 538.126683] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 538.132029] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 540.508370] Key type ._llcrypt registered [ 540.512545] Key type .llcrypt registered [ 540.610987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 540.620842] alg: No test for adler32 (adler32-zlib) [ 541.478212] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 544.819076] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 14:03:53 (1767035033) [ 544.899345] LNet: 52935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 544.907440] LNetError: 52935:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 544.920035] LNet: Removed LNI 10.1.2.3@tcp [ 545.225635] Key type .llcrypt unregistered [ 545.227504] Key type ._llcrypt unregistered [ 547.909366] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 548.015582] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 548.017859] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 548.021739] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 550.433360] Key type ._llcrypt registered [ 550.434979] Key type .llcrypt registered [ 550.530778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 550.541742] alg: No test for adler32 (adler32-zlib) [ 551.393542] LNet: 53237:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 551.405512] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 554.845150] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 14:04:03 (1767035043) [ 554.882690] LNet: 53819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 554.887775] LNetError: 53819:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 554.900854] LNet: Removed LNI 10.1.2.3@tcp [ 555.259586] Key type .llcrypt unregistered [ 555.260931] Key type ._llcrypt unregistered [ 557.720732] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 557.845599] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 557.848222] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 557.852349] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 560.261710] Key type ._llcrypt registered [ 560.263066] Key type .llcrypt registered [ 560.403784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 560.413694] alg: No test for adler32 (adler32-zlib) [ 561.251440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.300437] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 565.700222] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 14:04:14 (1767035054) [ 565.994811] LNet: 55052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 565.998963] LNetError: 55052:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 566.009685] LNet: Removed LNI 10.1.2.3@tcp [ 566.395716] Key type .llcrypt unregistered [ 566.401508] Key type ._llcrypt unregistered [ 569.256258] Key type ._llcrypt registered [ 569.257783] Key type .llcrypt registered [ 569.371808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 569.380874] alg: No test for adler32 (adler32-zlib) [ 570.228351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 570.241507] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 574.540445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 574.904931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_interrupt -r 1 [ 574.926544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_interrupt -r 1 [ 574.947701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 574.963259] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 575.105893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 575.133458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 575.157136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 575.177185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 575.196628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 575.217789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 575.239913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 575.257188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 575.267142] LNet: 55485:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 575.277381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 575.296662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 575.313736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 575.338564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 575.360478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 575.382501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 575.393302] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 575.408207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 575.673120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_dropped -r 1 [ 575.693721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_dropped -r 1 [ 575.720898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 575.731734] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 575.735368] LNet: Skipped 1 previous similar message [ 575.841519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 575.857097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 575.879656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 575.896545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 575.916394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 575.938354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 575.956883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 575.977100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 575.990554] LNet: 55594:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 576.000664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 576.018201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 576.038753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 576.067703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 576.090120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 576.109465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 576.133690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 576.404984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_aborted -r 1 [ 576.428776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_aborted -r 1 [ 576.451456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 576.609449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 576.636629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 576.660236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 576.679693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 576.699400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 576.718252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 576.739360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 576.760504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 576.778810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 576.801196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 576.820611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 576.837716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 576.858880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 576.880778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 576.901948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 577.222102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_no_route -r 1 [ 577.253510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_no_route -r 1 [ 577.278575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 577.289859] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 577.293308] LNet: Skipped 1 previous similar message [ 577.399844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 577.416317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 577.437603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 577.459507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 577.478506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 577.500640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 577.521679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 577.539831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 577.551716] LNet: 55812:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 577.554480] LNet: 55812:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 577.566146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 577.587700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 577.608569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 577.653556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 577.675222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 577.701745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 577.727755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 578.016082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_timeout -r 1 [ 578.035472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_timeout -r 1 [ 578.053683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 578.166667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 578.184285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 578.202196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 578.218962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 578.255291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 578.281113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 578.307401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 578.329810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 578.354821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 578.379839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 578.403925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 578.422358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 578.439419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 578.458329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 578.480409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 578.732516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_error -r 1 [ 578.755228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_error -r 1 [ 578.782053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 578.886552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 578.902987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 578.922253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 578.940238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 578.966800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 579.002955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 579.025739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 579.050673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 579.069101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 579.087476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 579.110782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 579.132369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 579.150463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 579.172157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 579.192033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 579.344987] LNet: 56070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 579.348772] LNetError: 56070:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 579.363613] LNet: Removed LNI 192.168.202.40@tcp [ 579.580412] Key type .llcrypt unregistered [ 579.581538] Key type ._llcrypt unregistered [ 587.218291] Key type ._llcrypt registered [ 587.219920] Key type .llcrypt registered [ 587.936369] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 14:04:36 (1767035076) [ 588.337036] Key type .llcrypt unregistered [ 588.339503] Key type ._llcrypt unregistered [ 591.184306] Key type ._llcrypt registered [ 591.186513] Key type .llcrypt registered [ 591.300372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 591.308957] alg: No test for adler32 (adler32-zlib) [ 592.144643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 592.166480] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 596.304909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 601.550760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.576590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 601.589916] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 602.310513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_interrupt -r 1 [ 602.336834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_interrupt -r 1 [ 602.357733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_interrupt -r 1 [ 602.384042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_interrupt -r 1 [ 602.413768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_interrupt -r 1 [ 602.438138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_interrupt -r 1 [ 602.467463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_interrupt -r 1 [ 602.490755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_interrupt -r 1 [ 602.510619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 602.524186] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 602.527397] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 602.636245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 602.652872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 602.671796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 602.687493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 602.706592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 602.726961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 602.745932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 602.767610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 602.780610] LNet: 57527:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 602.791512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 602.809317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 602.826946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 602.851957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 602.886706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 602.927988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 602.962492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 603.296751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_dropped -r 1 [ 603.321397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_dropped -r 1 [ 603.346933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_dropped -r 1 [ 603.372861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_dropped -r 1 [ 603.397682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_dropped -r 1 [ 603.428622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_dropped -r 1 [ 603.459744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_dropped -r 1 [ 603.489128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_dropped -r 1 [ 603.511374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 603.522511] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 603.529094] LNet: Skipped 2 previous similar messages [ 603.656679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 603.673858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 603.695864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 603.718851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 603.745586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 603.771972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 603.792851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 603.811571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 603.825802] LNet: 57648:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 603.839507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 603.875465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 603.897827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 603.923179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 603.940187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 603.964425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 603.991539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 604.303094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_aborted -r 1 [ 604.329801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_aborted -r 1 [ 604.361156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_aborted -r 1 [ 604.383579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_aborted -r 1 [ 604.405488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_aborted -r 1 [ 604.426197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_aborted -r 1 [ 604.451209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_aborted -r 1 [ 604.485174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_aborted -r 1 [ 604.507729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 604.523441] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 604.527958] LNet: Skipped 6 previous similar messages [ 604.685807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 604.710963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 604.734933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 604.753337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 604.771643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 604.793972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 604.817285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 604.835692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 604.849478] LNet: 57769:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 604.861864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 604.883965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 604.903977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 604.919181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 604.942684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 604.971742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 604.992851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 605.319778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_no_route -r 1 [ 605.349475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_no_route -r 1 [ 605.371139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_no_route -r 1 [ 605.390851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_no_route -r 1 [ 605.415662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_no_route -r 1 [ 605.433936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_no_route -r 1 [ 605.455725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_no_route -r 1 [ 605.478387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_no_route -r 1 [ 605.504687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 605.651804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 605.681209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 605.702695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 605.724036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 605.752573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 605.780732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 605.807199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 605.835825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 605.856776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 605.882519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 605.916703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 605.943437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 605.969631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 605.999694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 606.027728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 606.338242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_timeout -r 1 [ 606.364514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_timeout -r 1 [ 606.388322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_timeout -r 1 [ 606.410175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_timeout -r 1 [ 606.431570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_timeout -r 1 [ 606.456452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_timeout -r 1 [ 606.480562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_timeout -r 1 [ 606.505297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_timeout -r 1 [ 606.531147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 606.546718] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 606.551026] LNet: Skipped 3 previous similar messages [ 606.727610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 606.747052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 606.774170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 606.796187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 606.815942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 606.834041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 606.851229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 606.867507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 606.878767] LNet: 58011:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 606.883322] LNet: 58011:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 606.895520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 606.915551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 606.947493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 606.966171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 606.996696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 607.020800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 607.050078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 607.380203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_error -r 1 [ 607.406088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_error -r 1 [ 607.428924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_error -r 1 [ 607.462708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_error -r 1 [ 607.487433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_error -r 1 [ 607.520890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_error -r 1 [ 607.555574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_error -r 1 [ 607.591303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_error -r 1 [ 607.617935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 607.779357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 607.796817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 607.821843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 607.839624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 607.858751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 607.880736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 607.897677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 607.921896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 607.946628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 607.969554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 607.995921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 608.025206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 608.052645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 608.090318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 608.121253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 611.007735] LNet: 58207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.010489] LNetError: 58207:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 611.014656] LNet: Removed LNI 192.168.202.40@tcp [ 612.072728] LNet: Removed LNI 192.168.202.40@tcp1 [ 612.249905] Key type .llcrypt unregistered [ 612.251662] Key type ._llcrypt unregistered [ 620.692213] Key type ._llcrypt registered [ 620.693865] Key type .llcrypt registered [ 621.483110] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 14:05:10 (1767035110) [ 621.984588] Key type .llcrypt unregistered [ 621.986234] Key type ._llcrypt unregistered [ 625.209541] Key type ._llcrypt registered [ 625.211234] Key type .llcrypt registered [ 625.325798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 625.343197] alg: No test for adler32 (adler32-zlib) [ 626.163642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 626.176732] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 629.931568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 630.265816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_dropped -r 1 [ 630.286312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_dropped -r 1 [ 630.303172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 630.313470] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 630.414804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 630.431533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 630.451258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 630.475618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 630.495762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 630.515608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 630.535586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 630.553165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 630.562570] LNet: 59448:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 630.574927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 630.596810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 630.613134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 630.634093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 630.652731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 630.669956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 630.685636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 630.925830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_error -r 1 [ 630.946845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_error -r 1 [ 630.966440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 630.978097] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 631.080690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 631.097286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 631.113970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 631.130927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 631.150672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 631.167446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 631.192528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 631.215935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 631.229475] LNet: 59557:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 631.241607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 631.261966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 631.277886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 631.297469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 631.325662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 631.344944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 631.365792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 631.631168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_timeout -r 1 [ 631.649576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_timeout -r 1 [ 631.668035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 631.779810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 631.798815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 631.816937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 631.833113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 631.849533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 631.865937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 631.888224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 631.906677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 631.923672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 631.952720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 631.973840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 632.022739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 632.045276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 632.071094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 632.100970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 632.228606] LNet: 59707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.232449] LNetError: 59707:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 632.248382] LNet: Removed LNI 192.168.202.40@tcp [ 632.436730] Key type .llcrypt unregistered [ 632.438707] Key type ._llcrypt unregistered [ 640.019726] Key type ._llcrypt registered [ 640.021649] Key type .llcrypt registered [ 640.721631] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 14:05:29 (1767035129) [ 641.147380] Key type .llcrypt unregistered [ 641.149074] Key type ._llcrypt unregistered [ 643.935574] Key type ._llcrypt registered [ 643.937208] Key type .llcrypt registered [ 644.050757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 644.060767] alg: No test for adler32 (adler32-zlib) [ 644.908680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 644.928945] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 648.710796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 653.066966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.082566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 653.097150] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 653.814237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_dropped -r 1 [ 653.838241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e remote_dropped -r 1 [ 653.859558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_dropped -r 1 [ 653.880230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e remote_dropped -r 1 [ 653.900388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e remote_dropped -r 1 [ 653.921920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e remote_dropped -r 1 [ 653.944266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e remote_dropped -r 1 [ 653.962853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e remote_dropped -r 1 [ 653.982623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 654.001627] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 654.006132] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.140@tcp [ 654.119806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 654.140799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 654.166441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 654.185212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 654.205648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 654.224489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 654.243580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 654.261041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 654.272315] LNet: 61164:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 654.284576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 654.301375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 654.321987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 654.339872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 654.359898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 654.383601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 654.405986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 654.425854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 654.657110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_error -r 1 [ 654.675605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e remote_error -r 1 [ 654.696111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_error -r 1 [ 654.722607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e remote_error -r 1 [ 654.742766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e remote_error -r 1 [ 654.763720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e remote_error -r 1 [ 654.784638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e remote_error -r 1 [ 654.807121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e remote_error -r 1 [ 654.827812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 654.839673] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 654.843052] LNet: Skipped 2 previous similar messages [ 654.944225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 654.963536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 654.994776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 655.011919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 655.027726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 655.044375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 655.064730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 655.081955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 655.091263] LNet: 61288:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 655.105157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 655.126595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 655.146563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 655.163391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 655.181473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 655.199967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 655.218233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 655.234925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 655.507747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_timeout -r 1 [ 655.533888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e remote_timeout -r 1 [ 655.558484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_timeout -r 1 [ 655.579946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e remote_timeout -r 1 [ 655.602494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e remote_timeout -r 1 [ 655.628388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e remote_timeout -r 1 [ 655.653331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e remote_timeout -r 1 [ 655.676933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e remote_timeout -r 1 [ 655.699366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 655.836876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 655.852288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 655.868507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 655.887873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 655.908895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 655.926178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 655.945716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 655.965338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 655.986432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 656.008974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 656.034410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 656.051048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 656.074211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 656.094877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 656.119239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 656.137886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 658.948315] LNet: 61487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.953327] LNetError: 61487:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.957184] LNet: Removed LNI 192.168.202.40@tcp [ 658.963462] LNet: Removed LNI 192.168.202.40@tcp1 [ 659.163702] Key type .llcrypt unregistered [ 659.165460] Key type ._llcrypt unregistered [ 666.149514] Key type ._llcrypt registered [ 666.152076] Key type .llcrypt registered [ 666.787354] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 14:05:55 (1767035155) [ 666.996580] Key type .llcrypt unregistered [ 666.998266] Key type ._llcrypt unregistered [ 669.313444] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 669.363903] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 669.366280] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 669.368311] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 671.634573] Key type ._llcrypt registered [ 671.636198] Key type .llcrypt registered [ 671.705367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 671.714205] alg: No test for adler32 (adler32-zlib) [ 672.530852] LNet: 62555:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 672.541230] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 672.559789] LNet: 62569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.563423] LNetError: 62569:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.577871] LNet: Removed LNI 192.168.202.40@tcp [ 672.889562] Key type .llcrypt unregistered [ 672.891419] Key type ._llcrypt unregistered [ 675.357371] Key type ._llcrypt registered [ 675.359305] Key type .llcrypt registered [ 675.460528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 675.470850] alg: No test for adler32 (adler32-zlib) [ 676.297426] LNet: 62824:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 676.305854] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 676.307798] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 676.325826] LNet: 62838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.329814] LNetError: 62838:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.332796] LNet: Removed LNI 192.168.202.40@tcp [ 676.341793] LNet: Removed LNI 10.1.2.3@tcp1 [ 676.687690] Key type .llcrypt unregistered [ 676.689651] Key type ._llcrypt unregistered [ 679.231978] Key type ._llcrypt registered [ 679.233251] Key type .llcrypt registered [ 679.319198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 679.327681] alg: No test for adler32 (adler32-zlib) [ 680.174978] LNet: 63093:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 680.183466] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 680.184986] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 680.202474] LNet: 63107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.206832] LNetError: 63107:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 680.210914] LNet: Removed LNI 192.168.202.40@tcp [ 680.223830] LNet: Removed LNI 10.1.2.3@tcp [ 680.660346] Key type .llcrypt unregistered [ 680.661817] Key type ._llcrypt unregistered [ 683.239519] Key type ._llcrypt registered [ 683.241034] Key type .llcrypt registered [ 683.330974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 683.337928] alg: No test for adler32 (adler32-zlib) [ 684.156825] LNet: 63362:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 684.164570] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 684.166728] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 684.187277] LNet: 63376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.190407] LNetError: 63376:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 684.193227] LNet: Removed LNI 192.168.202.40@tcp [ 684.207089] LNet: Removed LNI 10.1.2.3@tcp [ 684.514655] Key type .llcrypt unregistered [ 684.516271] Key type ._llcrypt unregistered [ 687.267380] Key type ._llcrypt registered [ 687.268957] Key type .llcrypt registered [ 687.381763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 687.391775] alg: No test for adler32 (adler32-zlib) [ 688.218356] LNet: 63636:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 688.228554] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 688.230651] LNetError: 63636:0:(config.c:1560:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 688.233923] LNetError: Error -100 starting up LNI tcp [ 688.241803] LNet: Removed LNI 192.168.202.40@tcp [ 688.579457] Key type .llcrypt unregistered [ 688.581219] Key type ._llcrypt unregistered [ 693.569352] Key type ._llcrypt registered [ 693.570676] Key type .llcrypt registered [ 694.225887] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 14:06:23 (1767035183) [ 694.629523] Key type .llcrypt unregistered [ 694.632476] Key type ._llcrypt unregistered [ 697.057993] Key type ._llcrypt registered [ 697.059793] Key type .llcrypt registered [ 697.161590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 697.169491] alg: No test for adler32 (adler32-zlib) [ 697.998546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 698.013855] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 701.566655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 701.897095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e network_timeout -r 1 [ 701.917794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e network_timeout -r 1 [ 701.938227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 701.950642] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 702.077180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 702.092330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 702.107785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 702.112345] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 702.126311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 702.142211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 702.158979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 702.176922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 702.192649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 702.204879] LNet: 64884:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 702.219892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 702.238608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 702.259813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 702.279802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 702.300697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 702.333208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 702.485313] LNet: 64922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.495661] LNetError: 64922:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.526514] LNet: Removed LNI 192.168.202.40@tcp [ 703.738936] Key type .llcrypt unregistered [ 703.740754] Key type ._llcrypt unregistered [ 711.246321] Key type ._llcrypt registered [ 711.248741] Key type .llcrypt registered [ 711.339683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 711.347635] alg: No test for adler32 (adler32-zlib) [ 712.168561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 712.182694] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 715.812512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 719.538591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.557638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 719.569094] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 720.250696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e network_timeout -r 1 [ 720.272582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e network_timeout -r 1 [ 720.294696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e network_timeout -r 1 [ 720.318856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e network_timeout -r 1 [ 720.338979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e network_timeout -r 1 [ 720.362262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e network_timeout -r 1 [ 720.380143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e network_timeout -r 1 [ 720.398583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e network_timeout -r 1 [ 720.420202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 720.431083] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 720.536899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 720.555456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 720.570807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 720.587938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 720.605655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 720.623471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 720.642660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 720.660623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 720.671943] LNet: 65792:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 720.683444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 720.700988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 720.717338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 720.733163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 720.749251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 720.767643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 723.133716] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 723.270591] LNet: 65865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.273786] LNetError: 65865:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 723.277698] LNet: Removed LNI 192.168.202.40@tcp [ 723.286580] LNet: Removed LNI 192.168.202.40@tcp1 [ 723.481571] Key type .llcrypt unregistered [ 723.483182] Key type ._llcrypt unregistered [ 730.072932] Key type ._llcrypt registered [ 730.074259] Key type .llcrypt registered [ 730.719858] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 14:06:59 (1767035219) [ 730.897113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 730.906890] alg: No test for adler32 (adler32-zlib) [ 731.731471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.758701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 731.775635] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 732.159981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 732.170335] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 732.577567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 732.612912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 732.639657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 732.661617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 732.679472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 732.695569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 732.717732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.40@tcp [ 732.728414] LNetError: 66960:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.202.40@tcp to recovery [ 732.769311] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 732.769338] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 734.817991] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 734.907153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 738.915105] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 739.179753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 739.196968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 739.226290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 739.237593] LNet: 67067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.242661] LNetError: 67067:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 739.246065] LNet: Removed LNI 192.168.202.40@tcp [ 739.259587] LNet: Removed LNI 192.168.202.40@tcp1 [ 739.269389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.300965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 739.321594] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 739.684667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 739.695457] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 740.090795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 740.113253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 740.141795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 740.164562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 740.182848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 740.201683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 740.219696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 740.237932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.40@tcp [ 740.249698] LNetError: 67339:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.202.40@tcp to recovery [ 742.368403] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 742.372572] LNet: Skipped 1 previous similar message [ 743.467630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 746.464385] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 750.895937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 750.919200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 750.949580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 754.164553] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 14:07:23 (1767035243) [ 754.218646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.236160] LNet: 68056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.240615] LNetError: 68056:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 754.243283] LNetError: 68056:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 754.247063] LNet: Removed LNI 192.168.202.40@tcp [ 754.254640] LNet: Removed LNI 192.168.202.40@tcp1 [ 754.264195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.288363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 754.301248] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 754.685646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 754.698357] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 755.120807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 755.160323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 755.180448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 755.202510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 755.222412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 755.244354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 755.262238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.40@tcp --health 0 [ 755.297093] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.40@tcp [ 755.307814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 756.320445] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 756.323592] LNet: Skipped 1 previous similar message [ 770.950528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 771.028832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 771.052472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 771.075427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.40@tcp --health 500 [ 771.132437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 778.494404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 778.542542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 778.558473] LNet: 68564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.564674] LNetError: 68564:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 778.568215] LNetError: 68564:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 778.572796] LNet: Removed LNI 192.168.202.40@tcp [ 778.586400] LNet: Removed LNI 192.168.202.40@tcp1 [ 778.602975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.634769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 778.651388] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 779.080468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 779.093131] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 779.590359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 779.617879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 779.654355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 779.676659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 779.696873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 779.721611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 779.745721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 779.769787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.40@tcp --health 0 [ 780.704725] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 780.707867] LNet: Skipped 3 previous similar messages [ 780.974804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 791.538430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 791.555519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 791.569662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 794.678286] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 14:08:03 (1767035283) [ 794.876067] LNet: 69620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.883488] LNetError: 69620:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 794.886325] LNetError: 69620:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 794.889236] LNet: Removed LNI 192.168.202.40@tcp [ 794.898905] LNet: Removed LNI 192.168.202.40@tcp1 [ 795.203286] Key type .llcrypt unregistered [ 795.205020] Key type ._llcrypt unregistered [ 797.771364] Key type ._llcrypt registered [ 797.772970] Key type .llcrypt registered [ 797.877481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 797.886319] alg: No test for adler32 (adler32-zlib) [ 798.748344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 798.773835] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 803.263559] Lustre: DEBUG MARKER: Initial discovery [ 803.313464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.140@tcp [ 804.439397] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 804.521360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 804.541421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.140@tcp [ 804.552403] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 804.564454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 804.586576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.140@tcp -r 1 -m GET -l 3 [ 804.607683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.140@tcp -r 1 -m GET -e local_error [ 804.628160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.140@tcp -r 1 -m PUT -l 6 [ 804.960463] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 805.312289] Lustre: DEBUG MARKER: Force oleg240-client.virtnet to discover 192.168.202.140@tcp (in background) [ 805.372493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.140@tcp [ 806.062632] Lustre: DEBUG MARKER: Force oleg240-server to discover 192.168.202.40@tcp [ 807.283699] Lustre: DEBUG MARKER: Wait for 70371 [ 808.418981] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 809.215820] Lustre: DEBUG MARKER: Finished wait on 70371 [ 809.249196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 809.276595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 809.399766] LNet: 70827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.402644] LNetError: 70827:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.412777] LNet: Removed LNI 192.168.202.40@tcp [ 809.649703] Key type .llcrypt unregistered [ 809.651230] Key type ._llcrypt unregistered [ 816.365157] Key type ._llcrypt registered [ 816.366308] Key type .llcrypt registered [ 817.049323] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 14:08:26 (1767035306) [ 817.327753] Key type .llcrypt unregistered [ 817.329456] Key type ._llcrypt unregistered [ 819.887355] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 819.925589] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 819.927627] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 819.930028] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 822.244597] Key type ._llcrypt registered [ 822.246109] Key type .llcrypt registered [ 822.324444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 822.331719] alg: No test for adler32 (adler32-zlib) [ 823.151427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.180208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 823.194403] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 823.495899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 823.505295] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 827.202675] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 14:08:36 (1767035316) [ 827.276680] LNet: 72906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.281737] LNetError: 72906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.286775] LNet: Removed LNI 192.168.202.40@tcp [ 827.300902] LNet: Removed LNI 10.1.2.3@tcp [ 827.593529] Key type .llcrypt unregistered [ 827.595076] Key type ._llcrypt unregistered [ 830.164387] Key type ._llcrypt registered [ 830.165765] Key type .llcrypt registered [ 830.256231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 830.268054] alg: No test for adler32 (adler32-zlib) [ 831.090907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.117168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 831.130204] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 831.478756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 831.490657] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 831.971494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.40@tcp [ 832.993156] LNetError: Unexpected error -22 connecting to 192.168.202.40@tcp at host 192.168.202.40:988 [ 834.962688] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 14:08:44 (1767035324) [ 834.998700] LNet: 74027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.002275] LNetError: 74027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.005244] LNet: Removed LNI 192.168.202.40@tcp [ 835.017076] LNet: Removed LNI 10.1.2.3@tcp [ 835.298813] Key type .llcrypt unregistered [ 835.300499] Key type ._llcrypt unregistered [ 837.809108] Key type ._llcrypt registered [ 837.810698] Key type .llcrypt registered [ 837.891655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 837.901429] alg: No test for adler32 (adler32-zlib) [ 838.716777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.743495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 838.756750] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 839.134381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 839.148146] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 839.538979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.40@tcp1 --nid 192.168.202.40@tcp2 [ 841.551474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp1 192.168.202.40@tcp2 [ 841.562369] LNetError: 74865:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.40@tcp1 to 192.168.202.40@tcp2 - Route table may be misconfigured [ 841.578393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp2 192.168.202.40@tcp1 [ 844.241985] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 14:08:53 (1767035333) [ 844.300596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.310808] LNet: 75436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.313547] LNetError: 75436:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.316402] LNet: Removed LNI 192.168.202.40@tcp1 [ 845.352307] LNet: Removed LNI 192.168.202.40@tcp2 [ 845.370703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.397746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 845.413046] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 845.739685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 845.750403] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 846.126827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 846.147625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.40@tcp [ 846.164672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp1 192.168.202.40@tcp1 [ 846.183247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.40@tcp -d 192.168.202.40@tcp -e network_timeout [ 846.207189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -e network_timeout [ 846.227636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -e network_timeout [ 846.253503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -e network_timeout [ 846.275823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 846.290728] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 846.300379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 846.350213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 846.399237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 846.432487] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 849.119470] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 14:08:58 (1767035338) [ 849.168703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 849.180911] LNet: 76295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.184595] LNetError: 76295:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.188224] LNetError: 76295:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 849.195478] LNet: Removed LNI 192.168.202.40@tcp [ 849.203427] LNet: Removed LNI 192.168.202.40@tcp1 [ 849.217642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.252556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 849.521627] Key type .llcrypt unregistered [ 849.523350] Key type ._llcrypt unregistered [ 854.256934] Key type ._llcrypt registered [ 854.258392] Key type .llcrypt registered [ 854.873343] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 14:09:03 (1767035343) [ 855.047263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 855.054816] alg: No test for adler32 (adler32-zlib) [ 855.875456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.568576] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 857.329480] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 14:09:06 (1767035346) [ 857.399180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.410502] LNet: 77426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.430509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.459707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 857.473391] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 857.847191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 857.859451] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 858.288985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 858.309757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp1 [ 858.329107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp1 [ 861.167189] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 14:09:10 (1767035350) [ 862.026144] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg240-server [ 862.785656] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 14:09:11 (1767035351) [ 863.598223] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg240-server [ 864.438954] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 14:09:13 (1767035353) [ 865.363762] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg240-server [ 866.123819] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 14:09:15 (1767035355) [ 867.018848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg240-server [ 867.908401] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 14:09:17 (1767035357) [ 868.771616] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg240-server [ 869.585348] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 14:09:18 (1767035358) [ 870.511963] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg240-server [ 871.263669] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 14:09:20 (1767035360) [ 872.125235] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg240-server [ 872.926129] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 14:09:22 (1767035362) [ 873.752752] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg240-server [ 874.568354] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 14:09:23 (1767035363) [ 875.648979] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg240-server [ 876.547106] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 14:09:25 (1767035365) [ 876.645456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.673647] LNet: 81300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.678352] LNetError: 81300:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.682731] LNet: Removed LNI 192.168.202.40@tcp [ 877.738852] LNet: Removed LNI 192.168.202.40@tcp1 [ 877.761102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.800892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 877.818639] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 878.275655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 878.323921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 879.522785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.537460] LNet: 81457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.540867] LNetError: 81457:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.544879] LNetError: 81457:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 879.557921] LNet: Removed LNI 192.168.202.40@tcp [ 879.578639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.615971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 879.636865] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 880.096948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 880.228230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.255295] LNet: Removed LNI 192.168.202.40@tcp [ 881.273859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.319900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 881.346972] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 881.786824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 881.838162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 881.968701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.983713] LNet: 81750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.988460] LNet: 81750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 881.993937] LNetError: 81750:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.999469] LNetError: 81750:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 882.011707] LNet: Removed LNI 192.168.202.40@tcp [ 882.025712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.062924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 882.078348] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 882.634496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 882.782165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.817601] LNet: Removed LNI 192.168.202.40@tcp [ 883.829852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.867331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 883.884811] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 884.412982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 884.469181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 885.749178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.770166] LNetError: 82051:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.773424] LNetError: 82051:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 885.781230] LNet: Removed LNI 192.168.202.40@tcp [ 885.794696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.846251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 885.875513] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 886.448140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 886.658187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 886.677752] LNet: 82196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.681633] LNet: 82196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 887.724145] LNet: Removed LNI 192.168.202.40@tcp [ 887.752865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.791777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 887.806059] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 888.261441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 888.315994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 888.471952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.512513] LNet: Removed LNI 192.168.202.40@tcp [ 889.530391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.572293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 889.602555] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 890.113822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 890.294353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 890.307177] LNetError: 82489:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.310640] LNetError: 82489:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 890.322984] LNet: Removed LNI 192.168.202.40@tcp [ 890.334782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.371199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 890.386472] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 890.888411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 890.950977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 892.155833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.222419] LNet: Removed LNI 192.168.202.40@tcp [ 893.234838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.275080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 893.290731] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 893.769688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 893.954017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.016404] LNet: Removed LNI 192.168.202.40@tcp [ 895.037796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.080817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 895.094926] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 895.525236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 895.570854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 895.711806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.721768] LNet: 82938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.724857] LNet: 82938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 896.743487] LNet: Removed LNI 192.168.202.40@tcp [ 896.763106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.812278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 896.830315] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 897.222667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 897.422250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.476099] LNet: Removed LNI 192.168.202.40@tcp [ 898.520701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.583476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 898.604830] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 899.048296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 899.091595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.109452] LNetError: 83215:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.113750] LNetError: 83215:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 899.126295] LNet: Removed LNI 192.168.202.40@tcp [ 899.143979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.182322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 899.205879] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 899.632565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 903.719752] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 14:09:52 (1767035392) [ 903.786471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 903.810437] LNet: Removed LNI 192.168.202.40@tcp [ 903.827476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.880607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 903.898070] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/180] [ 904.673458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.707620] LNet: Removed LNI 192.168.202.40@tcp231 [ 904.721548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.767417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 904.790847] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/180] [ 904.794739] LNet: 84067:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 905.099894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 905.123751] LNet: Removed LNI 192.168.202.40@tcp231 [ 905.133819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 905.159400] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/180] [ 905.951764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.971604] LNet: Removed LNI 192.168.202.40@tcp231 [ 905.985624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.050924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 906.071248] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/0] [ 906.078481] LNet: 84219:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 906.343901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 906.369695] LNet: Removed LNI 192.168.202.40@tcp231 [ 906.380929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 906.398684] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/0] [ 907.056489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.079800] LNet: Removed LNI 192.168.202.40@tcp231 [ 907.092796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.134675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 907.151428] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/60] [ 907.155295] LNet: 84370:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 907.379732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 907.404834] LNet: Removed LNI 192.168.202.40@tcp231 [ 907.417884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 907.436184] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/60] [ 908.224959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.249504] LNet: Removed LNI 192.168.202.40@tcp231 [ 908.261531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.298347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 908.317600] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/180] [ 908.584939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 908.607177] LNet: Removed LNI 192.168.202.40@tcp231 [ 908.618382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 908.632341] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/180] [ 912.507997] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 14:10:01 (1767035401) [ 912.561548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.573142] LNet: 85226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.576419] LNet: 85226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 912.585619] LNet: Removed LNI 192.168.202.40@tcp231 [ 912.600342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.632967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 912.650110] LNet: Added LNI 192.168.202.40@tcp232 [8/256/0/180] [ 913.022930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 913.040713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 913.062107] LNet: Removed LNI 192.168.202.40@tcp232 [ 913.071046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 913.085515] LNet: Added LNI 192.168.202.40@tcp232 [8/256/0/180] [ 913.475712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 913.494332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 913.515566] LNet: Removed LNI 192.168.202.40@tcp232 [ 913.524340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 913.537976] LNet: Added LNI 192.168.202.40@tcp232 [8/256/0/180] [ 913.888996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 913.909243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 913.929192] LNet: Removed LNI 192.168.202.40@tcp232 [ 913.936800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 913.948738] LNet: Added LNI 192.168.202.40@tcp232 [8/256/0/180] [ 914.295918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 914.313561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 914.333457] LNet: Removed LNI 192.168.202.40@tcp232 [ 914.342583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 917.182623] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 14:10:06 (1767035406) [ 917.712912] Key type .llcrypt unregistered [ 917.715327] Key type ._llcrypt unregistered [ 920.096224] Key type ._llcrypt registered [ 920.097710] Key type .llcrypt registered [ 920.194258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 920.201201] alg: No test for adler32 (adler32-zlib) [ 921.025965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 921.042940] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 924.693051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 928.212371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.229857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 928.248075] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 928.849157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_interrupt -r 2 [ 928.868158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_interrupt -r 2 [ 928.887251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_interrupt -r 2 [ 928.903978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_interrupt -r 2 [ 928.921676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_interrupt -r 2 [ 928.941437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_interrupt -r 2 [ 928.962416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_interrupt -r 2 [ 928.982735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_interrupt -r 2 [ 929.005375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 929.017357] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 929.020613] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 929.123243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 929.138380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 929.155366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 929.171332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 929.188342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 929.203882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 929.219428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 929.235267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 929.246477] LNet: 87014:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 929.256244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 929.271417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 929.287481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 929.302748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 929.319341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 929.336467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 929.554532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_dropped -r 2 [ 929.572785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_dropped -r 2 [ 929.590290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_dropped -r 2 [ 929.609673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_dropped -r 2 [ 929.630282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_dropped -r 2 [ 929.648263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_dropped -r 2 [ 929.666461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_dropped -r 2 [ 929.685119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_dropped -r 2 [ 929.705415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 929.717047] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 929.825485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 929.841518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 929.860689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 929.876857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 929.893409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 929.910280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 929.927256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 929.943183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 929.951688] LNet: 87134:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 929.961191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 929.978218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 929.994386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 930.010605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 930.029371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 930.053678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 930.272222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_aborted -r 2 [ 930.294586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_aborted -r 2 [ 930.315224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_aborted -r 2 [ 930.337407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_aborted -r 2 [ 930.356699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_aborted -r 2 [ 930.375291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_aborted -r 2 [ 930.394477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_aborted -r 2 [ 930.413115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_aborted -r 2 [ 930.434739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 930.545035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 930.563747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 930.582656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 930.599763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 930.615885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 930.631620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 930.647796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 930.665583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 930.683651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 930.701292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 930.717640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 930.733656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 930.752802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 930.770558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 930.986115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_no_route -r 2 [ 931.003417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_no_route -r 2 [ 931.021388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_no_route -r 2 [ 931.037430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_no_route -r 2 [ 931.055932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_no_route -r 2 [ 931.073851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_no_route -r 2 [ 931.091200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_no_route -r 2 [ 931.109075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_no_route -r 2 [ 931.126611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 931.137798] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 931.140520] LNet: Skipped 2 previous similar messages [ 931.243221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 931.258398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 931.272485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 931.287157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 931.302723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 931.319612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 931.335483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 931.355274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 931.363668] LNet: 87374:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 931.366958] LNet: 87374:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 931.375979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 931.392489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 931.407366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 931.423361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 931.442330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 931.458133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 931.661120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_timeout -r 2 [ 931.680061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_timeout -r 2 [ 931.697387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_timeout -r 2 [ 931.714825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_timeout -r 2 [ 931.735659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_timeout -r 2 [ 931.754878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_timeout -r 2 [ 931.775540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_timeout -r 2 [ 931.794052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_timeout -r 2 [ 931.813383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 931.920280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 931.934950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 931.951924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 931.968241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 931.984112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 931.999230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 932.015643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 932.032963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 932.048467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 932.062555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 932.077804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 932.092960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 932.109348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 932.124990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 932.327474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_dropped -r 2 [ 932.345894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e remote_dropped -r 2 [ 932.363186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_dropped -r 2 [ 932.381601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e remote_dropped -r 2 [ 932.399844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e remote_dropped -r 2 [ 932.415965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e remote_dropped -r 2 [ 932.432637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e remote_dropped -r 2 [ 932.450478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e remote_dropped -r 2 [ 932.466588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 932.476646] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.140@tcp [ 932.562683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 932.577325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 932.592579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 932.612136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 932.628111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 932.643187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 932.661217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 932.675943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 932.691466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 932.707813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 932.724273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 932.739178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 932.756183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 932.772485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 935.162372] LNet: 87687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.165190] LNetError: 87687:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.169285] LNet: Removed LNI 192.168.202.40@tcp [ 935.175099] LNet: Removed LNI 192.168.202.40@tcp1 [ 935.352081] Key type .llcrypt unregistered [ 935.353774] Key type ._llcrypt unregistered [ 941.958783] Key type ._llcrypt registered [ 941.960139] Key type .llcrypt registered [ 942.563599] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 14:10:31 (1767035431) [ 942.747441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 942.755842] alg: No test for adler32 (adler32-zlib) [ 943.587692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.627430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 943.640044] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 944.100960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 944.120331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.40@tcp [ 944.135836] LNetError: 88660:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.202.40@tcp to recovery [ 944.187878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.40@tcp [ 944.255233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 947.614775] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 14:10:36 (1767035436) [ 947.683142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.699191] LNet: 89245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.703363] LNetError: 89245:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.719190] LNet: Removed LNI 192.168.202.40@tcp [ 947.744419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.776462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 947.792681] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 948.237792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 948.254724] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 948.785590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.40@tcp [ 948.801990] LNetError: 89515:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.202.40@tcp to recovery [ 949.792380] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 949.795272] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 949.934571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.40@tcp [ 955.099992] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 14:10:44 (1767035444) [ 956.020412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg240-server [ 956.842621] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 14:10:45 (1767035445) [ 956.888546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.901833] LNet: 90462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.907306] LNetError: 90462:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.914231] LNet: Removed LNI 192.168.202.40@tcp [ 956.921160] LNet: Removed LNI 192.168.202.40@tcp1 [ 956.936618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.966603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 956.981060] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 957.353948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 957.387764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 957.408795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 957.456436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 957.504212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 957.524758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 960.739217] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 14:10:49 (1767035449) [ 960.792472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 960.805770] LNet: 91187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.809871] LNetError: 91187:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 960.813201] LNetError: 91187:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 960.821658] LNet: Removed LNI 192.168.202.40@tcp [ 960.834271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.864727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 960.877758] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 964.262274] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 14:10:53 (1767035453) [ 964.321599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 964.337671] LNet: 91894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.348571] LNetError: 91894:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.359536] LNet: Removed LNI 192.168.202.40@tcp [ 964.371753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.396830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 964.410212] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 967.529316] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 14:10:56 (1767035456) [ 968.245803] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 969.098801] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 14:10:58 (1767035458) [ 969.332740] LNet: 92931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.336585] LNetError: 92931:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.346470] LNet: Removed LNI 192.168.202.40@tcp [ 969.639857] Key type .llcrypt unregistered [ 969.641571] Key type ._llcrypt unregistered [ 972.266876] Key type ._llcrypt registered [ 972.268435] Key type .llcrypt registered [ 972.366151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 972.375478] alg: No test for adler32 (adler32-zlib) [ 973.204316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 973.221200] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 977.128354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 979.064556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.140@tcp [ 980.263988] LNet: 93341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.267421] LNetError: 93341:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.279545] LNet: Removed LNI 192.168.202.40@tcp [ 980.483542] Key type .llcrypt unregistered [ 980.485173] Key type ._llcrypt unregistered [ 985.409590] Key type ._llcrypt registered [ 985.411045] Key type .llcrypt registered [ 986.254907] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 987.102573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 987.885797] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 14:11:17 (1767035477) [ 988.036326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 988.049905] alg: No test for adler32 (adler32-zlib) [ 988.884211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.925851] LNet: 94424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.221432] Key type .llcrypt unregistered [ 989.224061] Key type ._llcrypt unregistered [ 992.113568] Key type ._llcrypt registered [ 992.114980] Key type .llcrypt registered [ 992.218918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 992.228769] alg: No test for adler32 (adler32-zlib) [ 993.081689] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 993.083857] LNet: 94675:0:(config.c:1226:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 993.088284] LNet: 94675:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 993.121743] LNetError: 94687:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.43@tcp is being used as a gateway but routing feature is not turned on [ 993.131908] LNetError: 94687:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.202.43@tcp has gone from up to down [ 993.407198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 996.512168] LNetError: 94687:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.202.42@tcp has gone from up to down [ 996.516809] LNetError: 94687:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 996.523666] LNet: 94733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.527310] LNetError: 94733:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 996.536723] LNet: Removed LNI 192.168.202.40@tcp [ 996.548816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 996.582658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 996.596763] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 996.601779] LNet: 94741:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 996.607127] LNet: 94741:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 996.611918] LNetError: 94741:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.202.42@tcp is a gateway. Can not delete it [ 996.619314] LNetError: 94741:0:(peer.c:2217:lnet_del_peer_ni()) 192.168.202.42@tcp is a router. Can not be deleted [ 996.626450] LNet: 94741:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 996.627266] LNetError: 94738:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.45@tcp is being used as a gateway but routing feature is not turned on [ 996.638266] LNetError: 94738:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 999.264427] LNetError: 94743:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.202.41@tcp has gone from up to down [ 999.280095] LNetError: 94743:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 5 previous similar messages [ 999.643210] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 14:11:28 (1767035488) [ 1000.391648] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg240-server [ 1001.139829] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 14:11:30 (1767035490) [ 1001.984727] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg240-server [ 1002.826622] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 14:11:31 (1767035491) [ 1003.596163] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1005.328560] Key type lgssc registered [ 1005.881521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1014.937306] Key type lgssc unregistered [ 1015.104360] LNet: 97156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.107517] LNetError: 97156:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.123470] LNet: Removed LNI 192.168.202.40@tcp [ 1015.363961] Key type .llcrypt unregistered [ 1015.365643] Key type ._llcrypt unregistered [ 1021.668501] Key type ._llcrypt registered [ 1021.670189] Key type .llcrypt registered [ 1022.300604] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 14:11:51 (1767035511) [ 1022.453463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1022.462296] alg: No test for adler32 (adler32-zlib) [ 1023.286802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.322414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1023.345148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1023.366076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1023.391415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1026.409862] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 14:11:55 (1767035515) [ 1026.460311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1026.471943] LNet: 98552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.484895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1026.512657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1026.526846] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1026.946655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1026.961805] LNetError: 98701:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.974779] LNet: Removed LNI 192.168.202.40@tcp [ 1026.985369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.015432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1027.034294] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1027.273480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1027.288879] LNet: 98729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.292551] LNet: 98729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1027.302634] LNet: Removed LNI 192.168.202.40@tcp [ 1027.314726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.346318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1027.363425] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1030.566539] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 14:11:59 (1767035519) [ 1030.616934] LNet: 99313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.620046] LNetError: 99313:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.622975] LNetError: 99313:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1030.661817] LNet: Removed LNI 192.168.202.40@tcp [ 1030.965846] Key type .llcrypt unregistered [ 1030.967450] Key type ._llcrypt unregistered [ 1033.842410] Key type ._llcrypt registered [ 1033.843806] Key type .llcrypt registered [ 1033.955818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1033.963860] alg: No test for adler32 (adler32-zlib) [ 1034.802146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.859838] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1034.862386] LNet: *** cfs_fail_loc=e003, val=100*** [ 1035.317313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1035.333478] Ping reply received with mapping: [ 1035.334913] found metadata reply NI[192.168.202.40@tcp] [ 1035.336539] 28 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1035.338207] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1035.339838] 5a 5a 5a 5a 5a 5a 5a 5a [ 1035.438521] LNet: 99713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.444796] LNetError: 99713:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.454251] LNet: Removed LNI 192.168.202.40@tcp [ 1035.755436] Key type .llcrypt unregistered [ 1035.757201] Key type ._llcrypt unregistered [ 1041.383314] Key type ._llcrypt registered [ 1041.384701] Key type .llcrypt registered [ 1042.089608] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 14:12:11 (1767035531) [ 1042.356413] Key type .llcrypt unregistered [ 1042.359993] Key type ._llcrypt unregistered [ 1045.298335] Key type ._llcrypt registered [ 1045.301306] Key type .llcrypt registered [ 1045.407957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1045.415557] alg: No test for adler32 (adler32-zlib) [ 1046.372989] LNetError: 100746:0:(api-ni.c:2732:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1046.556600] Key type .llcrypt unregistered [ 1046.557955] Key type ._llcrypt unregistered [ 1049.074577] Key type ._llcrypt registered [ 1049.076066] Key type .llcrypt registered [ 1049.687710] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 14:12:18 (1767035538) [ 1049.919551] Key type .llcrypt unregistered [ 1049.922110] Key type ._llcrypt unregistered [ 1052.489276] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1052.492970] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1054.724112] Key type ._llcrypt registered [ 1054.725256] Key type .llcrypt registered [ 1054.803481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1054.810041] alg: No test for adler32 (adler32-zlib) [ 1055.628774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1055.661580] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1060.295472] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 14:12:29 (1767035549) [ 1060.350914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1060.361525] LNet: 102516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.367066] LNetError: 102516:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1060.376122] LNet: Removed LNI 10.1.2.3@tcp [ 1060.388261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.730773] Key type .llcrypt unregistered [ 1060.732060] Key type ._llcrypt unregistered [ 1063.227439] Key type ._llcrypt registered [ 1063.228863] Key type .llcrypt registered [ 1063.317393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1063.325469] alg: No test for adler32 (adler32-zlib) [ 1064.143775] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1064.173454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1064.241387] LNet: 102801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.244841] LNetError: 102801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.254705] LNet: Removed LNI 192.168.202.40@tcp [ 1064.535060] Key type .llcrypt unregistered [ 1064.538280] Key type ._llcrypt unregistered [ 1069.008928] Key type ._llcrypt registered [ 1069.010541] Key type .llcrypt registered [ 1069.660190] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 14:12:38 (1767035558) [ 1069.953484] Key type .llcrypt unregistered [ 1069.955323] Key type ._llcrypt unregistered [ 1072.596141] Key type ._llcrypt registered [ 1072.597667] Key type .llcrypt registered [ 1072.732874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1072.742971] alg: No test for adler32 (adler32-zlib) [ 1076.480482] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 14:12:45 (1767035565) [ 1076.540980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1076.566351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1076.596851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1076.611214] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1076.955765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1076.974210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1076.992679] LNet: Removed LNI 192.168.202.40@tcp [ 1077.110416] LNet: 104534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.316640] Key type .llcrypt unregistered [ 1077.318051] Key type ._llcrypt unregistered [ 1082.402144] Key type ._llcrypt registered [ 1082.403195] Key type .llcrypt registered [ 1082.991701] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 14:12:52 (1767035572) [ 1083.171833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1083.179950] alg: No test for adler32 (adler32-zlib) [ 1084.000711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1084.036994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1084.048867] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1084.387779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1084.410889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.40@tcp [ 1084.421779] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1084.424124] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1084.426293] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1084, 0, 0 [ 1084.429333] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1084.430731] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1084.433611] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1084.435627] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1084, 0, 0 [ 1084.438699] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1084.440382] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1084.442497] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1084.444601] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1084, 0, 1134 [ 1084.447729] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1084.449620] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1084.451680] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1084.453340] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1084, 0, 1134 [ 1084.455745] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1087.179991] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 14:12:56 (1767035576) [ 1087.366371] LNet: 106068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.369742] LNetError: 106068:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.381620] LNet: Removed LNI 192.168.202.40@tcp [ 1087.652382] Key type .llcrypt unregistered [ 1087.654190] Key type ._llcrypt unregistered [ 1090.085954] Key type ._llcrypt registered [ 1090.087548] Key type .llcrypt registered [ 1090.194183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1090.205516] alg: No test for adler32 (adler32-zlib) [ 1091.023126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1091.037053] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1094.799969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1098.586482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1098.605634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1098.621569] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1099.088860] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1099.168664] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1099.182456] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1099.184795] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1101.353207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1101.366279] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1106.482232] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 14:13:15 (1767035595) [ 1106.532588] LNet: 107562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.537093] LNetError: 107562:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.540306] LNet: Removed LNI 192.168.202.40@tcp [ 1107.552497] LNet: Removed LNI 192.168.202.40@tcp1 [ 1107.554217] LNetError: 107562:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1107.557232] LNetError: 107562:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1107.566584] LNet: Removed LNI 10.1.2.3@tcp99 [ 1107.872751] Key type .llcrypt unregistered [ 1107.873885] Key type ._llcrypt unregistered [ 1110.295191] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1110.297231] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1112.508735] Key type ._llcrypt registered [ 1112.509969] Key type .llcrypt registered [ 1112.592489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1112.601682] alg: No test for adler32 (adler32-zlib) [ 1113.420520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1113.449606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1113.464280] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1113.797316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1113.808732] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1114.842111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1114.865591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1114.890290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1114.900808] LNetError: 108308:0:(peer.c:2225:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1114.919252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1114.942827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1115.133825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1115.171173] LNet: 108328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.174466] LNetError: 108328:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.177359] LNet: Removed LNI 192.168.202.40@tcp [ 1115.187121] LNet: Removed LNI 10.1.2.3@tcp [ 1115.474472] Key type .llcrypt unregistered [ 1115.475946] Key type ._llcrypt unregistered [ 1120.208602] Key type ._llcrypt registered [ 1120.209798] Key type .llcrypt registered [ 1120.853958] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 14:13:29 (1767035609) [ 1121.039813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1121.046123] alg: No test for adler32 (adler32-zlib) [ 1121.861763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1121.886501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1121.896478] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1122.211157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1122.248471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg240-client@tcp [ 1125.134282] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 14:13:34 (1767035614) [ 1125.186101] LNet: 109849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.190398] LNetError: 109849:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.199927] LNet: Removed LNI 192.168.202.40@tcp [ 1125.471565] Key type .llcrypt unregistered [ 1125.473117] Key type ._llcrypt unregistered [ 1127.918661] Key type ._llcrypt registered [ 1127.920166] Key type .llcrypt registered [ 1127.999549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1128.006656] alg: No test for adler32 (adler32-zlib) [ 1128.822522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1128.836255] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1128.851693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1131.580165] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 14:13:40 (1767035620) [ 1132.245305] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1133.076899] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 14:13:42 (1767035622) [ 1133.795132] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1134.599445] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 14:13:43 (1767035623) [ 1134.637767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1134.651927] LNet: 111263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.655572] LNetError: 111263:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1134.666679] LNet: Removed LNI 192.168.202.40@tcp [ 1134.680663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1134.713693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1134.726350] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1135.079975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1135.105699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1137.934567] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 14:13:47 (1767035627) [ 1138.017730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1138.027892] LNet: 111977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.030861] LNetError: 111977:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1138.040452] LNet: Removed LNI 192.168.202.40@tcp [ 1138.052419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1138.080649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1138.097100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1138.383430] Key type .llcrypt unregistered [ 1138.384986] Key type ._llcrypt unregistered [ 1143.260321] Key type ._llcrypt registered [ 1143.261523] Key type .llcrypt registered [ 1143.896244] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 14:13:52 (1767035632) [ 1144.070068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1144.078538] alg: No test for adler32 (adler32-zlib) [ 1144.893493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1144.921607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1144.935091] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1145.267969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1145.294219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1148.146304] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 14:13:57 (1767035637) [ 1148.192798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1148.203870] LNet: 113522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.207834] LNetError: 113522:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.220134] LNet: Removed LNI 192.168.202.40@tcp [ 1148.233816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1148.266285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1148.283845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1150.869474] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 14:14:00 (1767035640) [ 1150.909091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1150.921164] LNet: 114102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1150.932104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1150.969059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1151.148132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1151.164457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1151.196543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1151.360748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1151.377581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1151.410722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1151.427983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1151.457802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1151.639725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1151.659623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1151.691721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1151.864453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1151.879702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1154.614568] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 14:14:03 (1767035643) [ 1154.681231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1154.697729] LNet: 114739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.700782] LNet: 114739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1154.712803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1154.739973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1154.754716] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1155.145971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1155.162322] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1155.568439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1155.594283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1155.704189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1155.715904] LNetError: 115019:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.719528] LNet: Removed LNI 192.168.202.40@tcp [ 1155.725534] LNet: Removed LNI 192.168.202.40@tcp2 [ 1155.741525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1155.779776] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1155.786574] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1155.790367] LNet: 115025:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1156.032372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1156.141780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1157.216169] LNet: Removed LNI 192.168.202.40@tcp [ 1157.217893] LNetError: 115063:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1157.221021] LNetError: 115063:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1157.230486] LNet: Removed LNI 192.168.202.40@tcp2 [ 1157.241551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1157.273549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1157.287786] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1157.671200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1157.682711] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1158.116242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1158.136883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1158.164335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1158.256960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1158.268532] LNet: 115347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1158.271387] LNet: 115347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1159.328186] LNet: Removed LNI 192.168.202.40@tcp [ 1159.329969] LNetError: 115347:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1159.334635] LNetError: 115347:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1159.345871] LNet: Removed LNI 192.168.202.40@tcp2 [ 1159.357490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1159.392078] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1159.395299] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1159.398302] LNet: 115353:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1159.622772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1159.710497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1159.720860] LNet: Removed LNI 192.168.202.40@tcp [ 1160.744075] LNet: Removed LNI 192.168.202.40@tcp2 [ 1160.755914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1160.790946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1160.804707] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1161.142230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1161.152996] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1161.565569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1161.592425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1161.694818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1162.720209] LNet: Removed LNI 192.168.202.40@tcp [ 1162.728382] LNet: Removed LNI 192.168.202.40@tcp2 [ 1162.737405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1162.769839] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1162.773788] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1162.777134] LNet: 115676:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1162.996165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1163.079922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1163.090804] LNet: 115713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.093653] LNet: 115713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1164.128220] LNet: Removed LNI 192.168.202.40@tcp [ 1164.130459] LNetError: 115713:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1164.134948] LNetError: 115713:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1164.144372] LNet: Removed LNI 192.168.202.40@tcp2 [ 1164.155149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1164.184170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1164.200466] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1164.536353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1164.547397] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1164.969417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1164.990362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1165.022209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1165.117432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1166.176206] LNet: Removed LNI 192.168.202.40@tcp [ 1167.206403] LNet: Removed LNI 192.168.202.40@tcp2 [ 1167.216618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1167.249352] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1167.252562] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1167.255461] LNet: 116003:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1167.464544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1167.542857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1167.553720] LNet: Removed LNI 192.168.202.40@tcp [ 1168.615229] LNet: Removed LNI 192.168.202.40@tcp2 [ 1168.628130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1168.660869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1168.672966] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1169.008244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1169.020334] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1169.407889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1169.424472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1169.453310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1169.633958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1170.656153] LNet: Removed LNI 192.168.202.40@tcp [ 1170.664465] LNet: Removed LNI 192.168.202.40@tcp2 [ 1170.678887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1170.721951] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1170.727663] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1170.948244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1171.109925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1171.122889] LNet: 116395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.127024] LNet: 116395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1172.192211] LNet: Removed LNI 192.168.202.40@tcp [ 1172.194089] LNetError: 116395:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.197272] LNetError: 116395:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1172.206578] LNet: Removed LNI 192.168.202.40@tcp2 [ 1172.216716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1172.250771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1172.265208] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1172.597708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1172.610651] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1173.008996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1173.026951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1173.043311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1173.072918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1173.236101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1174.304580] LNet: Removed LNI 192.168.202.40@tcp [ 1175.334368] LNet: Removed LNI 192.168.202.40@tcp2 [ 1175.345363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1175.376502] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1175.379963] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1175.383066] LNet: 116702:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1175.386448] LNet: 116702:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1175.603256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1178.759835] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1179.483667] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 14:14:28 (1767035668) [ 1179.531810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1179.543996] LNet: Removed LNI 192.168.202.40@tcp [ 1180.582907] LNet: Removed LNI 192.168.202.40@tcp2 [ 1180.593920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1180.716319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1180.727648] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1181.049655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1181.059554] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1181.445392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1181.455909] LNet: Removed LNI 192.168.202.40@tcp [ 1181.465480] LNet: Removed LNI 10.1.2.3@tcp [ 1181.473942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1181.501878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1181.513824] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1181.516435] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1181.698200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1181.712231] LNet: Removed LNI 192.168.202.40@tcp [ 1181.719725] LNet: Removed LNI 10.1.2.3@tcp [ 1184.215512] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 14:14:33 (1767035673) [ 1184.265735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1184.291322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1184.335429] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1184.394627] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1184.397030] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1184.399429] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1186.561622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1186.574125] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1186.885523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1186.895983] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1187.872446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1187.898597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1187.930330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.40@tcp -m PUT -e local_timeout -r 1 [ 1187.965200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1187.996121] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1187.998748] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1188.107801] LNetError: 118399:0:(lib-move.c:4947:lnet_parse()) 192.168.202.40@tcp, src 192.168.202.40@tcp: Dropping PUT (error -108 looking up sender) [ 1189.152572] LNet: 118856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1189.155666] LNet: 118856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1189.159208] LNetError: 118856:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1189.161874] LNetError: 118856:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1189.166330] LNet: Removed LNI 192.168.202.40@tcp [ 1189.175396] LNet: Removed LNI 10.1.2.3@tcp [ 1189.353307] Key type .llcrypt unregistered [ 1189.354948] Key type ._llcrypt unregistered [ 1193.960761] Key type ._llcrypt registered [ 1193.962398] Key type .llcrypt registered [ 1194.541584] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 14:14:43 (1767035683) [ 1194.701604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1194.708528] alg: No test for adler32 (adler32-zlib) [ 1195.539740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1195.740614] LNet: 119715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1196.054369] Key type .llcrypt unregistered [ 1196.055904] Key type ._llcrypt unregistered [ 1198.593432] Key type ._llcrypt registered [ 1198.594690] Key type .llcrypt registered [ 1198.676736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1198.683808] alg: No test for adler32 (adler32-zlib) [ 1199.502906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1199.518776] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1202.949427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1203.025361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1203.276282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.140@tcp [ 1203.374916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1203.390212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1203.405484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1203.420613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1203.435248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1203.450614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1203.466623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1203.483597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1203.493026] LNet: 120159:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1203.503167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1203.521820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1203.537540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1203.555232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1203.569812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1203.586104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1203.876245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1203.976410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1203.991433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1204.008504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1204.025277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1204.042218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1204.060795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1204.081437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1204.099235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1204.110932] LNet: 120276:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1204.119892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1204.137363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1204.154261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1204.168216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1204.184534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1204.202719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1204.463620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.140@tcp [ 1204.563464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1204.579251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1204.594934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1204.611082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1204.627388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1204.640286] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.140@tcp [ 1204.644208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1204.660220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1204.676774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1204.694404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1204.710507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1204.726357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1204.742756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1204.761436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1204.778807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1205.037667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1205.135302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1205.149682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1205.166433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1205.182868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1205.197595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1205.213034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1205.229692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1205.247438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1205.256880] LNet: 120510:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1205.261070] LNet: 120510:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1205.270703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1205.287353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1205.302173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1205.317537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1205.334583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1205.354132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1205.496068] LNet: 120555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1205.499457] LNetError: 120555:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1205.512470] LNet: Removed LNI 192.168.202.40@tcp [ 1205.687258] Key type .llcrypt unregistered [ 1205.688817] Key type ._llcrypt unregistered [ 1212.060437] Key type ._llcrypt registered [ 1212.061970] Key type .llcrypt registered [ 1212.658062] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 14:15:01 (1767035701) [ 1212.849097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1212.856578] alg: No test for adler32 (adler32-zlib) [ 1213.692525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1213.876591] LNet: 121437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1214.159396] Key type .llcrypt unregistered [ 1214.160946] Key type ._llcrypt unregistered [ 1216.585901] Key type ._llcrypt registered [ 1216.587258] Key type .llcrypt registered [ 1216.688511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1216.698322] alg: No test for adler32 (adler32-zlib) [ 1217.532479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1217.546845] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1221.087444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1225.160972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1225.180349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1225.191094] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1225.638903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1225.886987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.140@tcp [ 1225.982699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1226.006521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1226.022145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.040177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.056841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1226.073312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1226.088745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1226.104801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1226.116355] LNet: 122099:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1226.130413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.150924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.168519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.186782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1226.210211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.230594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1226.516263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1226.616700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1226.632787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1226.649737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.667365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.684889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1226.701224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1226.718245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1226.737868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1226.747298] LNet: 122216:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1226.758607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.777829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.784842] LNet: 2 peer NIs in recovery (showing 2): 192.168.202.140@tcp, 192.168.202.140@tcp1 [ 1226.797307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.821699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1226.841901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.865154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1227.174623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.140@tcp [ 1227.276331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1227.291515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1227.309455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1227.324700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1227.341947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1227.357551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1227.376321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1227.392900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1227.409687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1227.425854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1227.442843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1227.461401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1227.478669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.497839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1227.825679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1227.960782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1227.981897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1228.000854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.019867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.038788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.053885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.071439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1228.088546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1228.100038] LNet: 122451:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1228.102886] LNet: 122451:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1228.113509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1228.131964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1228.151646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.171793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.191223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.208479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1230.729169] LNet: 122530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1230.732602] LNetError: 122530:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1230.737467] LNet: Removed LNI 192.168.202.40@tcp [ 1230.752581] LNet: Removed LNI 192.168.202.40@tcp1 [ 1230.925828] Key type .llcrypt unregistered [ 1230.927500] Key type ._llcrypt unregistered [ 1237.270569] Key type ._llcrypt registered [ 1237.272128] Key type .llcrypt registered [ 1237.871852] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 14:15:26 (1767035726) [ 1240.838472] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 14:15:30 (1767035730) [ 1243.691242] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 14:15:32 (1767035732) [ 1246.627285] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 14:15:35 (1767035735) [ 1249.493582] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 14:15:38 (1767035738) [ 1252.413939] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 14:15:41 (1767035741) [ 1255.427088] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 14:15:44 (1767035744) [ 1258.401865] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 14:15:47 (1767035747) [ 1261.559637] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 14:15:50 (1767035750) [ 1264.135191] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 14:15:53 (1767035753) [ 1266.826320] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 14:15:56 (1767035756) [ 1269.832090] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 14:15:59 (1767035759) [ 1272.738711] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 14:16:01 (1767035761) [ 1275.561988] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 14:16:04 (1767035764) [ 1278.485457] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 14:16:07 (1767035767) [ 1281.477612] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 14:16:10 (1767035770) [ 1284.585846] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 14:16:13 (1767035773) [ 1287.564433] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 14:16:16 (1767035776) [ 1287.711879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1287.720189] alg: No test for adler32 (adler32-zlib) [ 1288.529785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1288.581141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1288.591323] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1289.044476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1289.053256] LNet: 133223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1289.056288] LNetError: 133223:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1289.069522] LNet: Removed LNI 192.168.202.40@tcp [ 1289.079571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1289.100841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1289.110993] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1291.865189] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 14:16:21 (1767035781) [ 1294.822267] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 14:16:24 (1767035784) [ 1297.703899] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 14:16:26 (1767035786) [ 1300.754786] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 14:16:29 (1767035789) [ 1303.318483] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 14:16:32 (1767035792) [ 1305.993332] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 14:16:35 (1767035795) [ 1308.680841] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 14:16:37 (1767035797) [ 1311.388174] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 14:16:40 (1767035800) [ 1314.345634] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 14:16:43 (1767035803) [ 1317.481689] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 14:16:46 (1767035806) [ 1320.432561] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 14:16:49 (1767035809) [ 1323.330825] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 14:16:52 (1767035812) [ 1326.211941] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1208 sec ========== 14:16:55 (1767035815) [ 1326.327128] LNet: 140680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.330683] LNetError: 140680:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1326.340185] LNet: Removed LNI 192.168.202.40@tcp [ 1326.632629] Key type .llcrypt unregistered [ 1326.634257] Key type ._llcrypt unregistered [ 1331.240873] Key type ._llcrypt registered [ 1331.245592] Key type .llcrypt registered [ 1331.451969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1331.459598] alg: No test for adler32 (adler32-zlib) [ 1332.368971] Lustre: Lustre: Build Version: 2.17.0_RC3_1_ga0d6154 [ 1332.495750] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1334.096132] Key type lgssc registered [ 1334.600558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1360.901435] Lustre: Mounted lustre-client [ 1363.655803] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1371.472824] LustreError: 143122:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff96f0089aa800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1371.481656] LustreError: 143122:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1371.498634] Lustre: Unmounted lustre-client [ 1385.027845] Key type lgssc unregistered [ 1385.162566] LNet: 143602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1385.165533] LNetError: 143602:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1385.176336] LNet: Removed LNI 192.168.202.40@tcp [ 1385.458181] Key type .llcrypt unregistered [ 1385.459655] Key type ._llcrypt unregistered