[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 451629589 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002295] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.007318] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009006] pid_max: default: 32768 minimum: 301 [ 0.010119] LSM: Security Framework initializing [ 0.011032] Yama: becoming mindful. [ 0.012022] SELinux: Initializing. [ 0.013046] *** VALIDATE selinux *** [ 0.020624] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024691] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025129] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026080] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027094] *** VALIDATE tmpfs *** [ 0.028401] *** VALIDATE proc *** [ 0.029193] *** VALIDATE cgroup *** [ 0.030005] *** VALIDATE cgroup2 *** [ 0.031221] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032131] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034023] Spectre V2 : User space: Vulnerable [ 0.035005] Speculative Store Bypass: Vulnerable [ 0.038038] debug: unmapping init [mem 0xffffffffad659000-0xffffffffad660fff] [ 0.041000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041612] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042021] ... version: 2 [ 0.043007] ... bit width: 48 [ 0.044008] ... generic registers: 4 [ 0.045008] ... value mask: 0000ffffffffffff [ 0.046008] ... max period: 00007fffffffffff [ 0.047008] ... fixed-purpose events: 3 [ 0.047977] ... event mask: 000000070000000f [ 0.048363] rcu: Hierarchical SRCU implementation. [ 0.050656] smp: Bringing up secondary CPUs ... [ 0.051631] x86: Booting SMP configuration: [ 0.052023] .... node #0, CPUs: #1 #2 #3 [ 0.060085] smp: Brought up 1 node, 4 CPUs [ 0.062008] smpboot: Max logical packages: 1 [ 0.063013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.089060] node 0 deferred pages initialised in 24ms [ 0.092305] devtmpfs: initialized [ 0.093280] x86/mm: Memory block size: 128MB [ 0.096052] gcov: version magic: 0x41383552 [ 0.098297] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.099071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.100225] pinctrl core: initialized pinctrl subsystem [ 0.101161] [ 0.101638] ************************************************************* [ 0.102007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.103007] ** ** [ 0.104009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.105008] ** ** [ 0.106007] ** This means that this kernel is built to expose internal ** [ 0.107007] ** IOMMU data structures, which may compromise security on ** [ 0.108009] ** your system. ** [ 0.109008] ** ** [ 0.110012] ** If you see this message and you are not debugging the ** [ 0.111012] ** kernel, report this immediately to your vendor! ** [ 0.112008] ** ** [ 0.113008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.114008] ************************************************************* [ 0.116522] NET: Registered protocol family 16 [ 0.118011] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.119054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.120048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.121550] cpuidle: using governor menu [ 0.122790] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.125605] PCI: Using configuration type 1 for base access [ 0.128121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.135219] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.136013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.137454] cryptd: max_cpu_qlen set to 1000 [ 0.139200] ACPI: Added _OSI(Module Device) [ 0.140010] ACPI: Added _OSI(Processor Device) [ 0.142009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.143009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.149548] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.155588] ACPI: Interpreter enabled [ 0.157153] ACPI: PM: (supports S0 S3 S4 S5) [ 0.159010] ACPI: Using IOAPIC for interrupt routing [ 0.161126] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.165490] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.178342] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.180045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.184026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.187117] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.192693] acpiphp: Slot [2] registered [ 0.193135] acpiphp: Slot [5] registered [ 0.194134] acpiphp: Slot [6] registered [ 0.196169] acpiphp: Slot [3] registered [ 0.197153] acpiphp: Slot [4] registered [ 0.199106] acpiphp: Slot [7] registered [ 0.201162] acpiphp: Slot [8] registered [ 0.203119] acpiphp: Slot [9] registered [ 0.204294] acpiphp: Slot [10] registered [ 0.206246] acpiphp: Slot [11] registered [ 0.208130] acpiphp: Slot [12] registered [ 0.210166] acpiphp: Slot [13] registered [ 0.212160] acpiphp: Slot [14] registered [ 0.213117] acpiphp: Slot [15] registered [ 0.215084] acpiphp: Slot [16] registered [ 0.216085] acpiphp: Slot [17] registered [ 0.218093] acpiphp: Slot [18] registered [ 0.219074] acpiphp: Slot [19] registered [ 0.220036] acpiphp: Slot [20] registered [ 0.221068] acpiphp: Slot [21] registered [ 0.222082] acpiphp: Slot [22] registered [ 0.223141] acpiphp: Slot [23] registered [ 0.224089] acpiphp: Slot [24] registered [ 0.226250] acpiphp: Slot [25] registered [ 0.227093] acpiphp: Slot [26] registered [ 0.229152] acpiphp: Slot [27] registered [ 0.231110] acpiphp: Slot [28] registered [ 0.232189] acpiphp: Slot [29] registered [ 0.234125] acpiphp: Slot [30] registered [ 0.236126] acpiphp: Slot [31] registered [ 0.238102] PCI host bridge to bus 0000:00 [ 0.239020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.241064] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.243024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.245034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.248029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.250067] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.252344] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.255403] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.260629] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.268024] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.272143] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.275028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.303039] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.306014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.310543] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.313105] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.315042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.318961] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.321011] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.328012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.333011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.338423] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.342012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.347015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.365018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.393000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.406012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.416018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.433014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.448277] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.452548] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.456448] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.460638] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.464316] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.470221] iommu: Default domain type: Passthrough [ 0.471000] SCSI subsystem initialized [ 0.471000] ACPI: bus type USB registered [ 0.473487] usbcore: registered new interface driver usbfs [ 0.476081] usbcore: registered new interface driver hub [ 0.477114] usbcore: registered new device driver usb [ 0.479185] pps_core: LinuxPPS API ver. 1 registered [ 0.481008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.487115] PTP clock support registered [ 0.489431] EDAC MC: Ver: 3.0.0 [ 0.490169] PCI: Using ACPI for IRQ routing [ 0.492602] NetLabel: Initializing [ 0.494007] NetLabel: domain hash size = 128 [ 0.495014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.496096] NetLabel: unlabeled traffic allowed by default [ 0.497330] vgaarb: loaded [ 0.499257] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.500034] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.504301] clocksource: Switched to clocksource kvm-clock [ 0.719511] VFS: Disk quotas dquot_6.6.0 [ 0.722675] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.727656] *** VALIDATE ramfs *** [ 0.729422] *** VALIDATE hugetlbfs *** [ 0.731704] pnp: PnP ACPI init [ 0.734636] pnp: PnP ACPI: found 6 devices [ 0.755248] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.760895] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.763961] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.767065] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.770126] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.772757] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.775204] NET: Registered protocol family 2 [ 0.778610] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.784441] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.789294] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.795760] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.800381] TCP: Hash tables configured (established 65536 bind 65536) [ 0.805912] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.809785] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.813217] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.816592] NET: Registered protocol family 1 [ 0.819399] RPC: Registered named UNIX socket transport module. [ 0.822024] RPC: Registered udp transport module. [ 0.823983] RPC: Registered tcp transport module. [ 0.825807] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.828669] NET: Registered protocol family 44 [ 0.830818] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.833600] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.836276] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.839496] PCI: CLS 0 bytes, default 64 [ 0.841718] Unpacking initramfs... [ 3.036858] debug: unmapping init [mem 0xffff92b63cc64000-0xffff92b63ffcffff] [ 3.041593] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.043530] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.046204] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.712647] Initialise system trusted keyrings [ 3.714034] Key type blacklist registered [ 3.715675] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.724427] zbud: loaded [ 3.727656] *** VALIDATE nfs *** [ 3.728645] *** VALIDATE nfs4 *** [ 3.730245] pstore: using deflate compression [ 3.733873] Platform Keyring initialized [ 3.939200] NET: Registered protocol family 38 [ 3.941253] Key type asymmetric registered [ 3.942982] Asymmetric key parser 'x509' registered [ 3.944881] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.949741] io scheduler mq-deadline registered [ 3.953586] io scheduler kyber registered [ 3.955823] io scheduler bfq registered [ 3.959675] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.963614] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.966858] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.969603] ACPI: Power Button [PWRF] [ 3.976702] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.989278] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.007831] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.045301] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.074971] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.080278] Non-volatile memory driver v1.3 [ 4.082426] Linux agpgart interface v0.103 [ 4.114292] virtio_blk virtio1: [vda] 140360 512-byte logical blocks (71.9 MB/68.5 MiB) [ 4.117971] vda: detected capacity change from 0 to 71864320 [ 4.160672] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.174454] vdb: detected capacity change from 0 to 1073741824 [ 4.187740] libphy: Fixed MDIO Bus: probed [ 4.193828] usbcore: registered new interface driver usbserial_generic [ 4.204377] usbserial: USB Serial support registered for generic [ 4.218914] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.228923] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.231284] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.235639] mousedev: PS/2 mouse device common for all mice [ 4.240358] rtc_cmos 00:05: RTC can wake from S4 [ 4.241459] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.257980] rtc_cmos 00:05: registered as rtc0 [ 4.260076] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.261591] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.270682] intel_pstate: CPU model not supported [ 4.274498] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.278092] hid: raw HID events driver (C) Jiri Kosina [ 4.287419] usbcore: registered new interface driver usbhid [ 4.289722] usbhid: USB HID core driver [ 4.291877] drop_monitor: Initializing network drop monitor service [ 4.294540] Initializing XFRM netlink socket [ 4.296517] NET: Registered protocol family 10 [ 4.300357] Segment Routing with IPv6 [ 4.301676] NET: Registered protocol family 17 [ 4.304330] mpls_gso: MPLS GSO support [ 4.310419] RAS: Correctable Errors collector initialized. [ 4.312577] AVX version of gcm_enc/dec engaged. [ 4.314704] AES CTR mode by8 optimization enabled [ 4.488653] sched_clock: Marking stable (4488498182, 0)->(5534433357, -1045935175) [ 4.492541] registered taskstats version 1 [ 4.494952] Loading compiled-in X.509 certificates [ 4.500970] zswap: loaded using pool lzo/zbud [ 4.531187] Key type big_key registered [ 4.545236] Key type encrypted registered [ 4.546319] ima: No TPM chip found, activating TPM-bypass! [ 4.547560] ima: Allocated hash algorithm: sha1 [ 4.548948] ima: No architecture policies found [ 4.550341] evm: Initialising EVM extended attributes: [ 4.552323] evm: security.selinux [ 4.553530] evm: security.ima [ 4.554637] evm: security.capability [ 4.555888] evm: HMAC attrs: 0x1 [ 4.557572] rtc_cmos 00:05: setting system clock to 2026-04-29 00:02:16 UTC (1777420936) [ 4.565460] debug: unmapping init [mem 0xffffffffae603000-0xffffffffae7fffff] [ 4.569519] debug: unmapping init [mem 0xffffffffad382000-0xffffffffad658fff] [ 4.582154] Write protecting the kernel read-only data: 28672k [ 4.588429] debug: unmapping init [mem 0xffffffffaba03000-0xffffffffabbfffff] [ 4.592516] debug: unmapping init [mem 0xffffffffac314000-0xffffffffac3fffff] [ 4.630291] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.640318] systemd[1]: Detected virtualization kvm. [ 4.642854] systemd[1]: Detected architecture x86-64. [ 4.646340] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.687804] systemd[1]: No hostname configured. [ 4.689793] systemd[1]: Set hostname to . [ 4.692021] random: systemd: uninitialized urandom read (16 bytes read) [ 4.693851] systemd[1]: Initializing machine ID from random generator. [ 4.764462] random: ln: uninitialized urandom read (6 bytes read) [ 4.908558] random: systemd: uninitialized urandom read (16 bytes read) [ 4.911411] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.922379] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.927456] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.886919] device-mapper: uevent: version 1.0.3 [ 5.888768] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.010567] virtio_net virtio0 ens2: renamed from eth0 [ 7.302356] scsi host0: ata_piix [ 7.455206] scsi host1: ata_piix [ 7.458996] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.462310] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 10.989156] random: fast init done [ 12.061735] random: crng init done [ 12.062939] random: 7 urandom warning(s) missed due to ratelimiting [ 12.765379] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.453577] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.333485] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.816317] SELinux: Disabled at runtime. [ 16.908084] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.919225] systemd[1]: Detected virtualization kvm. [ 16.921474] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.016398] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.019960] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.025093] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.029087] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.032166] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.043958] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.067253] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 18.200754] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 19.239399] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.158520] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.209519] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.513118] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.610188] EDAC sbridge: Ver: 1.1.2 [ 22.572869] Key type dns_resolver registered [ 23.301906] NFS: Registering the id_resolver key type [ 23.306138] Key type id_resolver registered [ 23.309079] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg246-client login: [ 55.350208] libcfs: loading out-of-tree module taints kernel. [ 55.457368] Key type ._llcrypt registered [ 55.460568] Key type .llcrypt registered [ 55.660098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 55.665719] alg: No test for adler32 (adler32-zlib) [ 56.662892] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 56.974184] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 58.584137] Key type lgssc registered [ 59.151107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 108.385546] Lustre: Mounted lustre-client [ 111.028937] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 123.817749] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing check_logdir /tmp/testlogs/ [ 125.469642] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing yml_node [ 127.319337] Lustre: DEBUG MARKER: Client: 2.17.50.194 [ 128.256165] Lustre: DEBUG MARKER: MDS: 2.17.50.194 [ 129.225439] Lustre: DEBUG MARKER: OSS: 2.17.50.194 [ 129.813085] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Apr 28 20:04:21 EDT 2026 [ 134.112192] Lustre: lustre-OST0000-osc-ffff92b6881cb800: disconnect after 25s idle [ 135.014125] LustreError: 7904:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff92b6881cb800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 135.027533] Lustre: Unmounted lustre-client [ 148.022304] Key type lgssc unregistered [ 148.193819] LNet: 8351:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 148.197637] LNetError: 8351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 148.207512] LNet: Removed LNI 192.168.202.46@tcp [ 148.524151] Key type .llcrypt unregistered [ 148.525564] Key type ._llcrypt unregistered [ 158.410216] Key type ._llcrypt registered [ 158.411783] Key type .llcrypt registered [ 158.497957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 158.505384] alg: No test for adler32 (adler32-zlib) [ 159.326353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 159.363923] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 159.373630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 161.059322] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 161.120466] LNet: 9228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 161.124599] LNetError: 9228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 161.131341] LNet: Removed LNI 192.168.202.46@tcp [ 161.383392] Key type .llcrypt unregistered [ 161.385073] Key type ._llcrypt unregistered [ 163.538883] Key type ._llcrypt registered [ 163.540633] Key type .llcrypt registered [ 164.125440] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 20:04:55 (1777421095) [ 164.261712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 164.268240] alg: No test for adler32 (adler32-zlib) [ 165.092844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 165.129474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 165.140937] LNet: 9638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 165.151741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 165.186353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 165.198676] LNet: 9645:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 167.890790] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 20:04:59 (1777421099) [ 167.920754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 167.931434] LNet: 10216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 167.941113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 167.976869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 170.835803] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 20:05:02 (1777421102) [ 170.867289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 170.880281] LNet: 10801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 170.892938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 170.932521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 173.449868] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 20:05:04 (1777421104) [ 173.522450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 173.534157] LNet: 11389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 173.548865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 173.587156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 176.080448] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 20:05:07 (1777421107) [ 176.123572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 176.141603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 176.175680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 176.359082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 178.910543] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 20:05:10 (1777421110) [ 178.973939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 178.984659] LNet: 12561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 178.987672] LNet: 12561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 178.996847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 179.025893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 179.172600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 179.333323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 181.722703] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 20:05:13 (1777421113) [ 181.765560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 181.790512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 181.827858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 184.075769] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 20:05:15 (1777421115) [ 184.115339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 184.133901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 184.158340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 184.171508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 184.319814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 184.335901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 184.496298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 184.511573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 184.668459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 184.684927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 184.855812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 184.876408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 185.035514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 185.049066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 185.193984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 185.209862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 185.361685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 185.376843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 185.533542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 185.549379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 185.712617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 185.727083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 185.884940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 185.900344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 186.059474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 186.075799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 186.221094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 186.243153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 188.902232] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 20:05:20 (1777421120) [ 188.978319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 188.987599] LNet: 14422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 188.990422] LNet: 14422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 189.002200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 189.035488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 189.051559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 191.726434] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 20:05:23 (1777421123) [ 191.769089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 191.785967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 191.817321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 191.834041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 194.588947] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 20:05:25 (1777421125) [ 194.644777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 194.666594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 194.702616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 194.722560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 197.355613] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 20:05:28 (1777421128) [ 197.408053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 197.429551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 197.468197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 197.484199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 200.271027] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 20:05:31 (1777421131) [ 200.304864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 200.327482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 200.366590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 200.385550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 203.381151] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 20:05:34 (1777421134) [ 203.426069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 203.453708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 203.501367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 203.520548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 206.246754] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 20:05:37 (1777421137) [ 206.321764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 206.331610] LNet: 17937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 206.334698] LNet: 17937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 206.345713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 206.380947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 206.548280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 209.040244] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 20:05:40 (1777421140) [ 209.074473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 209.094063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 209.177324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 209.266908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 211.834561] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 20:05:43 (1777421143) [ 211.866728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 211.886082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 211.938578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 212.029353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 214.348351] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 20:05:45 (1777421145) [ 214.382031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 214.404633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.585450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 214.691311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 217.277528] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 20:05:48 (1777421148) [ 217.313024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 217.334878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 217.368281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 217.388885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 219.876725] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 20:05:51 (1777421151) [ 219.909078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 219.928164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 219.962376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 219.982356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 222.553348] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 20:05:54 (1777421154) [ 222.585839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.609415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.641983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 224.996726] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 20:05:56 (1777421156) [ 225.037758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 225.064501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 225.096222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 227.474699] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 20:05:58 (1777421158) [ 227.528174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 227.545980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 227.577845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 227.727695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 230.128315] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 20:06:01 (1777421161) [ 230.178255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 230.196402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.228932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 230.244932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 232.712428] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 20:06:04 (1777421164) [ 232.752956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 232.769872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 232.803696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 232.821927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 235.372256] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 20:06:06 (1777421166) [ 235.413628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.434495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.468072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 235.482312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 237.779839] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 20:06:09 (1777421169) [ 237.820640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 237.837812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 237.871345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 237.888673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 240.134649] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 20:06:11 (1777421171) [ 240.175930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.185556] LNet: 25710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 240.187860] LNet: 25710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 240.200542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.224926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 240.249734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 240.258274] LNetError: 25724:0:(peer.c:2167:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 240.275888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 240.291298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 242.387315] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 20:06:13 (1777421173) [ 242.419625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 242.440484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 242.480699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MJjbkDJMFO [ 242.513650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MJjbkDJMFO [ 242.538345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MJjbkDJMFO [ 244.671158] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 20:06:16 (1777421176) [ 244.697260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.713372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.738791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 244.752770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 244.772607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 246.897753] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 20:06:18 (1777421178) [ 246.928658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 246.945734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 246.975626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 248.039017] LNet: Message forwarding will be enabled in 2s [ 250.080123] LNet: Message forwarding enabled [ 250.175381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 250.186411] LNet: Message forwarding will stop in 2s to 6s [ 254.339739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 254.776745] LNet: Message forwarding disabled [ 255.266436] LNet: Message forwarding will be enabled in 2s [ 257.547935] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 20:06:28 (1777421188) [ 257.590430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.177059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.209355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 260.362619] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 20:06:31 (1777421191) [ 260.396879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 260.414240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.439325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 260.453244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 260.466044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 260.478875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 260.492725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 260.506764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 260.519889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 260.534517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 260.552675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 260.567265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 260.582636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 260.595201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 260.607445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 260.619649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 260.633517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 260.647132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 260.661546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 260.678160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 260.692493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 260.701895] LNetError: 28781:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 262.775102] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 20:06:34 (1777421194) [ 262.810427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 262.826512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 262.858629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 265.040992] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 20:06:36 (1777421196) [ 265.743414] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg246-server [ 266.385378] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 20:06:37 (1777421197) [ 266.437124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 266.455936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 266.478943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 266.489242] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 267.061556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[47-51]@tcp [ 267.071337] LNet: 30495:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 267.084194] LNetError: 30284:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.50@tcp is being used as a gateway but routing feature is not turned on [ 267.090586] LNetError: 30284:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.50@tcp has gone from up to down [ 268.133088] LNetError: 30284:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.50@tcp is being used as a gateway but routing feature is not turned on [ 268.137778] LNetError: 30284:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 269.157256] LNetError: 30284:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.50@tcp is being used as a gateway but routing feature is not turned on [ 269.162418] LNetError: 30284:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 269.551241] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 20:06:40 (1777421200) [ 270.176533] LNetError: 30290:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.49@tcp has gone from up to down [ 270.182613] LNetError: 30290:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 270.238531] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg246-server [ 270.804729] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 20:06:42 (1777421202) [ 270.854803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 270.864508] LNetError: 31428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 270.876501] LNet: Removed LNI 192.168.202.46@tcp [ 270.884849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 270.907281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 270.918276] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 271.160173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[47-52/2]@tcp [ 271.168685] LNet: 31545:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 271.172366] LNet: 31545:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 271.179717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[47-52/2]@tcp [ 273.391673] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 20:06:44 (1777421204) [ 273.423776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.432515] LNetError: 32132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 273.440666] LNet: Removed LNI 192.168.202.46@tcp [ 273.447940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.480434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 273.503573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.519170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.547375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 273.569926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.584776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.605379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 273.635858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.651081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.679029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 273.710542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.730755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.755443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 273.784780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.799172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.826118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 273.856947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.869462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.890898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 273.920260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.934784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.961769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 273.991199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 274.005326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 274.026444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 274.056314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 274.072046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 274.096581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 274.127150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 274.143918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 274.167428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 274.191463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 274.209164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 274.240894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 276.308146] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 20:06:47 (1777421207) [ 276.913921] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg246-server [ 277.513462] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 20:06:48 (1777421208) [ 278.191886] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg246-server [ 278.791134] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 20:06:50 (1777421210) [ 279.088326] Key type .llcrypt unregistered [ 279.089659] Key type ._llcrypt unregistered [ 281.054178] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 281.137731] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 281.139992] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 281.144076] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 283.376082] Key type ._llcrypt registered [ 283.377437] Key type .llcrypt registered [ 283.443837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 283.450869] alg: No test for adler32 (adler32-zlib) [ 284.254968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 284.281057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 284.292192] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 284.609602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 284.623339] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 285.484392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 285.496341] LNet: Removed LNI 10.1.2.3@tcp [ 285.540814] LNet: 34256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 285.544731] LNetError: 34256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 285.553503] LNet: Removed LNI 192.168.202.46@tcp [ 285.806530] Key type .llcrypt unregistered [ 285.808145] Key type ._llcrypt unregistered [ 289.746424] Key type ._llcrypt registered [ 289.747741] Key type .llcrypt registered [ 290.255316] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 20:07:01 (1777421221) [ 290.427282] Key type .llcrypt unregistered [ 290.428761] Key type ._llcrypt unregistered [ 292.358794] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 292.399731] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 292.404360] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 292.406415] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 294.656738] Key type ._llcrypt registered [ 294.657738] Key type .llcrypt registered [ 294.720753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 294.727672] alg: No test for adler32 (adler32-zlib) [ 295.532443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.557873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 295.570517] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 295.825401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 295.834170] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 296.684928] LNet: 35746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 296.688089] LNetError: 35746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 296.691322] LNet: Removed LNI 192.168.202.46@tcp [ 296.698284] LNet: Removed LNI 10.1.2.3@tcp [ 296.921544] Key type .llcrypt unregistered [ 296.923047] Key type ._llcrypt unregistered [ 300.699525] Key type ._llcrypt registered [ 300.700881] Key type .llcrypt registered [ 301.217691] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 20:07:12 (1777421232) [ 301.424302] Key type .llcrypt unregistered [ 301.425447] Key type ._llcrypt unregistered [ 303.588109] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 303.627324] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 303.629657] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 303.631805] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 305.864110] Key type ._llcrypt registered [ 305.865653] Key type .llcrypt registered [ 305.947434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 305.954497] alg: No test for adler32 (adler32-zlib) [ 306.777116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.803847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 306.815635] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 307.583615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 307.593601] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 308.415922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 308.426723] LNet: Removed LNI 10.1.2.3@tcp [ 308.434414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 308.455893] LNet: Removed LNI 10.1.2.31@tcp [ 308.463829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 308.474782] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 308.768916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 308.778028] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 309.159833] LNet: 37664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.162968] LNetError: 37664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 309.166329] LNet: Removed LNI 10.1.2.3@tcp [ 309.177240] LNet: Removed LNI 10.1.2.31@tcp [ 309.436417] Key type .llcrypt unregistered [ 309.437734] Key type ._llcrypt unregistered [ 313.326330] Key type ._llcrypt registered [ 313.327642] Key type .llcrypt registered [ 313.864659] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 20:07:25 (1777421245) [ 314.001495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 314.007938] alg: No test for adler32 (adler32-zlib) [ 314.827708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.855716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 314.869117] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 315.219198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 315.231291] LNet: 38615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 315.234944] LNetError: 38615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 315.244407] LNet: Removed LNI 192.168.202.46@tcp [ 315.256133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.282333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.46@tcp [ 315.293308] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 315.634539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.46@tcp [ 315.659368] LNet: Removed LNI 192.168.202.46@tcp [ 315.963406] Key type .llcrypt unregistered [ 315.964770] Key type ._llcrypt unregistered [ 319.868616] Key type ._llcrypt registered [ 319.870258] Key type .llcrypt registered [ 320.409844] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 20:07:31 (1777421251) [ 321.085263] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg246-server [ 321.725246] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 20:07:33 (1777421253) [ 321.907952] Key type .llcrypt unregistered [ 321.909459] Key type ._llcrypt unregistered [ 323.944936] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 324.024181] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 324.025968] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 324.028770] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 326.245453] Key type ._llcrypt registered [ 326.246902] Key type .llcrypt registered [ 326.317935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 326.325044] alg: No test for adler32 (adler32-zlib) [ 327.131504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.157790] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 327.437064] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 328.256297] LNet: 40583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.259551] LNetError: 40583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 328.262444] LNet: Removed LNI 192.168.202.46@tcp1 [ 328.276403] LNet: Removed LNI 10.1.2.3@tcp2 [ 328.509342] Key type .llcrypt unregistered [ 328.510529] Key type ._llcrypt unregistered [ 332.366675] Key type ._llcrypt registered [ 332.368225] Key type .llcrypt registered [ 332.900411] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 20:07:44 (1777421264) [ 333.071596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 333.079493] alg: No test for adler32 (adler32-zlib) [ 333.881879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.901672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 333.910078] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 334.186788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.194083] LNet: 41550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.196908] LNetError: 41550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 334.210460] LNet: Removed LNI 192.168.202.46@tcp [ 334.218471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.242148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 334.252148] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 336.652788] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 20:07:48 (1777421268) [ 336.683190] LNet: 42133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.686255] LNetError: 42133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 336.692695] LNet: Removed LNI 192.168.202.46@tcp [ 336.939585] Key type .llcrypt unregistered [ 336.940794] Key type ._llcrypt unregistered [ 339.073847] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 339.152913] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 339.154466] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 339.157056] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 341.337889] Key type ._llcrypt registered [ 341.339321] Key type .llcrypt registered [ 341.407286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 341.414223] alg: No test for adler32 (adler32-zlib) [ 342.225628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.250899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 342.261699] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 342.547311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 342.555787] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 343.373375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.382620] LNet: 42879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.385518] LNetError: 42879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 343.388705] LNet: Removed LNI 192.168.202.46@tcp [ 343.395502] LNet: Removed LNI 10.1.2.3@tcp [ 343.404329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.431708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 343.444566] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 343.447599] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 343.627679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.639607] LNet: Removed LNI 192.168.202.46@tcp [ 343.645381] LNet: Removed LNI 10.1.2.3@tcp [ 343.654591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.681350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 343.692179] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 343.694770] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 343.864178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 343.875391] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 344.289660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.298954] LNet: 43068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.301779] LNet: 43068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 344.305218] LNetError: 43068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 344.310198] LNetError: 43068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 344.313576] LNet: Removed LNI 192.168.202.46@tcp [ 344.315142] LNet: Removed LNI 10.1.2.3@tcp [ 344.324110] LNet: Removed LNI 192.168.202.46@tcp2 [ 344.335186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.360664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 344.370799] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 344.372573] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 344.374962] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 346.931345] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 20:07:58 (1777421278) [ 346.987922] LNet: 43662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.991635] LNetError: 43662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 346.994267] LNetError: 43662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 346.997641] LNet: Removed LNI 192.168.202.46@tcp [ 346.998873] LNet: Removed LNI 10.1.2.3@tcp [ 347.007060] LNet: Removed LNI 192.168.202.46@tcp2 [ 347.262648] Key type .llcrypt unregistered [ 347.263964] Key type ._llcrypt unregistered [ 349.559215] Key type ._llcrypt registered [ 349.560916] Key type .llcrypt registered [ 349.673781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 349.682183] alg: No test for adler32 (adler32-zlib) [ 350.491771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.514951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 350.525697] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 350.880936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.891144] LNet: 44059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.894959] LNetError: 44059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 350.903529] LNet: Removed LNI 192.168.202.46@tcp [ 350.912498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.939281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 350.950637] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 351.145512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.162380] LNet: Removed LNI 192.168.202.46@tcp [ 351.172449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.203828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 351.218528] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 351.403106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.414417] LNet: 44119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.417581] LNet: 44119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 351.423905] LNetError: 44119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 351.427450] LNetError: 44119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 351.443355] LNet: Removed LNI 192.168.202.46@tcp [ 351.453351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.480120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 351.490829] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 351.656820] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 351.718358] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 351.731652] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 351.734502] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 353.873185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.883584] LNet: 44198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.886485] LNetError: 44198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 353.899060] LNet: Removed LNI 192.168.202.46@tcp [ 353.913639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.943529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 353.956262] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 356.748794] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 20:08:08 (1777421288) [ 356.804246] LNet: 44790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.807836] LNetError: 44790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 356.819138] LNet: Removed LNI 192.168.202.46@tcp [ 357.109880] Key type .llcrypt unregistered [ 357.111678] Key type ._llcrypt unregistered [ 359.398465] Key type ._llcrypt registered [ 359.400207] Key type .llcrypt registered [ 359.538626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 359.563189] alg: No test for adler32 (adler32-zlib) [ 360.397254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.422396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 360.433504] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 360.814305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.823991] LNet: 45192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.827121] LNetError: 45192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 360.837886] LNet: Removed LNI 192.168.202.46@tcp [ 360.847759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.876144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 360.888620] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 363.548780] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 20:08:14 (1777421294) [ 363.583031] LNet: 45774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.588733] LNetError: 45774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 363.597206] LNet: Removed LNI 192.168.202.46@tcp [ 363.898778] Key type .llcrypt unregistered [ 363.900046] Key type ._llcrypt unregistered [ 365.957944] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 366.008895] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 366.011481] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 366.013952] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 368.233513] Key type ._llcrypt registered [ 368.234935] Key type .llcrypt registered [ 368.299949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 368.306292] alg: No test for adler32 (adler32-zlib) [ 369.114412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.137623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 369.149345] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 369.481960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.491163] LNet: 46232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.493944] LNetError: 46232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 369.511340] LNet: Removed LNI 192.168.202.46@tcp [ 369.520903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.546646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 369.558574] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 372.040493] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 20:08:23 (1777421303) [ 372.079084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.090443] LNet: 46827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.093478] LNetError: 46827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 372.102624] LNet: Removed LNI 192.168.202.46@tcp [ 372.116648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.160549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 374.402258] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 20:08:25 (1777421305) [ 374.435607] LNet: 47404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.704498] Key type .llcrypt unregistered [ 374.705922] Key type ._llcrypt unregistered [ 376.937267] Key type ._llcrypt registered [ 376.938644] Key type .llcrypt registered [ 377.004674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 377.010731] alg: No test for adler32 (adler32-zlib) [ 377.820938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.844901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 377.856632] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 378.196151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.205164] LNet: 47809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.207740] LNetError: 47809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 378.221772] LNet: Removed LNI 192.168.202.46@tcp [ 378.230707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.261201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 378.272198] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 380.631870] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 20:08:32 (1777421312) [ 380.663895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.675095] LNet: 48396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.678281] LNetError: 48396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 380.686637] LNet: Removed LNI 192.168.202.46@tcp [ 380.698724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.730477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 382.984044] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 20:08:34 (1777421314) [ 383.549287] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 384.170796] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 20:08:35 (1777421315) [ 384.723172] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 385.323968] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 20:08:36 (1777421316) [ 385.829022] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 386.438895] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 20:08:37 (1777421317) [ 387.029357] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 387.650299] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 20:08:39 (1777421319) [ 388.264769] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 388.864080] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 20:08:40 (1777421320) [ 388.893654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.903168] LNet: 50445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.917112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.945647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 388.955731] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 389.245802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.46 [ 389.258545] LNet: 50584:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 389.263472] LNet: Removed LNI 192.168.202.46@tcp [ 391.397099] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 20:08:42 (1777421322) [ 391.445329] LNet: 51149:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.449069] LNetError: 51149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 391.457390] LNet: Removed LNI 192.168.202.46@tcp [ 391.686685] Key type .llcrypt unregistered [ 391.687903] Key type ._llcrypt unregistered [ 393.826596] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 393.926388] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 393.928563] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 393.932376] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 396.240460] Key type ._llcrypt registered [ 396.241979] Key type .llcrypt registered [ 396.300588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 396.306314] alg: No test for adler32 (adler32-zlib) [ 397.301747] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 399.663119] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 20:08:51 (1777421331) [ 399.703095] LNet: 52051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 399.705983] LNetError: 52051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 399.724824] LNet: Removed LNI 10.1.2.3@tcp [ 400.002498] Key type .llcrypt unregistered [ 400.004273] Key type ._llcrypt unregistered [ 402.107862] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 402.216184] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 402.218851] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 402.221696] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 404.543424] Key type ._llcrypt registered [ 404.544786] Key type .llcrypt registered [ 404.603340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 404.609972] alg: No test for adler32 (adler32-zlib) [ 405.438915] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 407.970295] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 20:08:59 (1777421339) [ 408.013439] LNet: 52943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.016657] LNetError: 52943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 408.028353] LNet: Removed LNI 10.1.2.3@tcp [ 408.284596] Key type .llcrypt unregistered [ 408.286174] Key type ._llcrypt unregistered [ 410.489927] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 410.594681] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 410.596185] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 410.597869] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 412.919071] Key type ._llcrypt registered [ 412.920543] Key type .llcrypt registered [ 412.999782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 413.006367] alg: No test for adler32 (adler32-zlib) [ 413.826785] LNet: 53249:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 413.833320] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 416.344606] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 20:09:07 (1777421347) [ 416.381202] LNet: 53833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 416.384772] LNetError: 53833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 416.392977] LNet: Removed LNI 10.1.2.3@tcp [ 416.658554] Key type .llcrypt unregistered [ 416.659956] Key type ._llcrypt unregistered [ 418.717934] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 418.823488] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 418.825203] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 418.827371] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 421.125785] Key type ._llcrypt registered [ 421.127968] Key type .llcrypt registered [ 421.209438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 421.215808] alg: No test for adler32 (adler32-zlib) [ 422.027997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.065580] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 425.247919] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 20:09:16 (1777421356) [ 425.406032] LNet: 55074:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.410531] LNetError: 55074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 425.431547] LNet: Removed LNI 10.1.2.3@tcp [ 425.679543] Key type .llcrypt unregistered [ 425.681590] Key type ._llcrypt unregistered [ 427.803468] Key type ._llcrypt registered [ 427.804950] Key type .llcrypt registered [ 427.889071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 427.895940] alg: No test for adler32 (adler32-zlib) [ 428.710109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 428.721098] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 431.757222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 431.983499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_interrupt -r 1 [ 432.010650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_interrupt -r 1 [ 432.025166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 432.033743] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 432.120317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 432.133429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 432.148077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 432.162438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 432.176865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 432.191563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 432.204806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 432.218052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 432.228078] LNet: 55510:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 432.236190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 432.249938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 432.264445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 432.280168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 432.296502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 432.311674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 432.325186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 432.508250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_dropped -r 1 [ 432.524330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_dropped -r 1 [ 432.540854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 432.550849] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 432.634736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 432.649258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 432.662159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 432.675743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 432.689635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 432.702642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 432.716499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 432.731342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 432.740218] LNet: 55619:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 432.749514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 432.764365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 432.780842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 432.794539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 432.800091] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 432.810382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 432.826657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 432.844276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 433.034295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_aborted -r 1 [ 433.049090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_aborted -r 1 [ 433.066424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 433.151947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 433.165233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 433.177527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 433.189353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 433.203831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 433.218421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 433.232647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 433.246809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 433.262039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 433.277251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 433.293402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 433.307260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 433.323696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 433.339430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 433.353132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 433.539987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_no_route -r 1 [ 433.558075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_no_route -r 1 [ 433.575740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 433.584568] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 433.587326] LNet: Skipped 1 previous similar message [ 433.667734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 433.680974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 433.695745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 433.713545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 433.727876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 433.741761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 433.756509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 433.770669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 433.779434] LNet: 55837:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 433.782758] LNet: 55837:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 433.791244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 433.805880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 433.819734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 433.832613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 433.847304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 433.862026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 433.877102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 434.061665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_timeout -r 1 [ 434.079862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_timeout -r 1 [ 434.096679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 434.186869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 434.201410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 434.216178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 434.229595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 434.243302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 434.256410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 434.270118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 434.284398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 434.298764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 434.312765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 434.326992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 434.340799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 434.354425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 434.369365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 434.383790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 434.582697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_error -r 1 [ 434.598325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_error -r 1 [ 434.613882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 434.699901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 434.713072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 434.726879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 434.740677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 434.754056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 434.768501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 434.781282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 434.795739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 434.809991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 434.825401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 434.840730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 434.855826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 434.870231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 434.884039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 434.899440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 434.995709] LNet: 56097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.998294] LNetError: 56097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 435.005473] LNet: Removed LNI 192.168.202.46@tcp [ 435.163646] Key type .llcrypt unregistered [ 435.165581] Key type ._llcrypt unregistered [ 440.621021] Key type ._llcrypt registered [ 440.622169] Key type .llcrypt registered [ 441.082981] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 20:09:32 (1777421372) [ 441.382462] Key type .llcrypt unregistered [ 441.383748] Key type ._llcrypt unregistered [ 443.339511] Key type ._llcrypt registered [ 443.340697] Key type .llcrypt registered [ 443.403085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 443.408688] alg: No test for adler32 (adler32-zlib) [ 444.209658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 444.221433] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 447.067869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 450.381622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.397144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 450.407253] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 450.918615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_interrupt -r 1 [ 450.933829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_interrupt -r 1 [ 450.949104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_interrupt -r 1 [ 450.965646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_interrupt -r 1 [ 450.983755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_interrupt -r 1 [ 451.001268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_interrupt -r 1 [ 451.019362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_interrupt -r 1 [ 451.034767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_interrupt -r 1 [ 451.050564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 451.058103] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 451.060696] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 451.142828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 451.156351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 451.171162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 451.187667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 451.201785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 451.217290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 451.231576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 451.247077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 451.255842] LNet: 57558:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 451.265881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 451.281754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 451.303206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 451.319601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 451.334093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 451.348934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 451.364297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 451.579379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_dropped -r 1 [ 451.599388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_dropped -r 1 [ 451.618496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_dropped -r 1 [ 451.636688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_dropped -r 1 [ 451.654717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_dropped -r 1 [ 451.673805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_dropped -r 1 [ 451.691849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_dropped -r 1 [ 451.709893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_dropped -r 1 [ 451.730864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 451.739894] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 451.742429] LNet: Skipped 2 previous similar messages [ 451.824154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 451.839498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 451.855684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 451.881134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 451.898774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 451.917183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 451.934681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 451.952684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 451.963533] LNet: 57679:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 451.971749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 451.994899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 452.011979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 452.040859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 452.064193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 452.082400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 452.102867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 452.299288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_aborted -r 1 [ 452.315825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_aborted -r 1 [ 452.332773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_aborted -r 1 [ 452.349614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_aborted -r 1 [ 452.366382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_aborted -r 1 [ 452.385175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_aborted -r 1 [ 452.404024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_aborted -r 1 [ 452.421422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_aborted -r 1 [ 452.441427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 452.539083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 452.555046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 452.574960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 452.590474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 452.604525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 452.618482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 452.634494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 452.650508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 452.666711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 452.681187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 452.700703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 452.715688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 452.732670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 452.751037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 452.770524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 452.966364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_no_route -r 1 [ 452.983369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_no_route -r 1 [ 453.001407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_no_route -r 1 [ 453.019821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_no_route -r 1 [ 453.036473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_no_route -r 1 [ 453.055151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_no_route -r 1 [ 453.071688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_no_route -r 1 [ 453.088565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_no_route -r 1 [ 453.108921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 453.119519] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 453.121911] LNet: Skipped 6 previous similar messages [ 453.205968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 453.221177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 453.239883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 453.254967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 453.270854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 453.284814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 453.300130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 453.316862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 453.325706] LNet: 57921:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 453.329291] LNet: 57921:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 453.340632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 453.357218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 453.369781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 453.383785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 453.398363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 453.414065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 453.432640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 453.647210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_timeout -r 1 [ 453.662490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_timeout -r 1 [ 453.677767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_timeout -r 1 [ 453.695187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_timeout -r 1 [ 453.712739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_timeout -r 1 [ 453.730428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_timeout -r 1 [ 453.753813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_timeout -r 1 [ 453.770333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_timeout -r 1 [ 453.789221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 453.875377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 453.888553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 453.901732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 453.917095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 453.930417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 453.945884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 453.962463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 453.977575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 453.992374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 454.006962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 454.021590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 454.036649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 454.051746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 454.066866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 454.082513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 454.268623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_error -r 1 [ 454.286128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_error -r 1 [ 454.303507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_error -r 1 [ 454.319614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_error -r 1 [ 454.336127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_error -r 1 [ 454.351581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_error -r 1 [ 454.369381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_error -r 1 [ 454.389236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_error -r 1 [ 454.407606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 454.502769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 454.516711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 454.530842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 454.544777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 454.558945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 454.571908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 454.586120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 454.602496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 454.620143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 454.634536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 454.650022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 454.666129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 454.680860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 454.695725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 454.714642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 456.957982] LNet: 58239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 456.963318] LNetError: 58239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 456.967789] LNet: Removed LNI 192.168.202.46@tcp [ 458.022515] LNet: Removed LNI 192.168.202.46@tcp1 [ 458.186585] Key type .llcrypt unregistered [ 458.188476] Key type ._llcrypt unregistered [ 464.067778] Key type ._llcrypt registered [ 464.069701] Key type .llcrypt registered [ 464.650550] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 20:09:55 (1777421395) [ 464.948532] Key type .llcrypt unregistered [ 464.950073] Key type ._llcrypt unregistered [ 467.151391] Key type ._llcrypt registered [ 467.152911] Key type .llcrypt registered [ 467.237527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 467.246107] alg: No test for adler32 (adler32-zlib) [ 468.055782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 468.069629] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 471.284239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 471.537475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e remote_dropped -r 1 [ 471.555571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e remote_dropped -r 1 [ 471.572079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 471.581940] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 471.676756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 471.692965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 471.707748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 471.723194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 471.736949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 471.752672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 471.769312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 471.784914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 471.794330] LNet: 59485:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 471.804038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 471.818793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 471.834265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 471.847530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 471.861692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 471.877672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 471.893644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 472.087732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e remote_error -r 1 [ 472.104573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e remote_error -r 1 [ 472.125098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 472.134250] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 472.223063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 472.237334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 472.252558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 472.267433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 472.281762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 472.295918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 472.310235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 472.323308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 472.332452] LNet: 59594:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 472.340594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 472.358152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 472.372674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 472.386958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 472.401769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 472.416511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 472.431630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 472.635074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e remote_timeout -r 1 [ 472.651318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e remote_timeout -r 1 [ 472.669565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 472.756364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 472.770874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 472.783936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 472.798333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 472.814649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 472.827424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 472.840452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 472.855235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 472.869695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 472.884799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 472.898929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 472.912892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 472.929903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 472.945412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 472.961144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 473.071334] LNet: 59744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 473.074375] LNetError: 59744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 473.090449] LNet: Removed LNI 192.168.202.46@tcp [ 473.248507] Key type .llcrypt unregistered [ 473.250194] Key type ._llcrypt unregistered [ 478.746136] Key type ._llcrypt registered [ 478.747495] Key type .llcrypt registered [ 479.239206] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 20:10:10 (1777421410) [ 479.510414] Key type .llcrypt unregistered [ 479.512273] Key type ._llcrypt unregistered [ 481.589337] Key type ._llcrypt registered [ 481.590632] Key type .llcrypt registered [ 481.684634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 481.691443] alg: No test for adler32 (adler32-zlib) [ 482.496374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 482.512765] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 485.693247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 488.927836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.942734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 488.957098] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 489.505476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e remote_dropped -r 1 [ 489.522871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e remote_dropped -r 1 [ 489.538451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e remote_dropped -r 1 [ 489.554747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e remote_dropped -r 1 [ 489.570224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e remote_dropped -r 1 [ 489.585527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e remote_dropped -r 1 [ 489.601468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e remote_dropped -r 1 [ 489.617620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e remote_dropped -r 1 [ 489.633618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 489.642424] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 489.645509] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.146@tcp [ 489.735147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 489.751086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 489.765820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 489.780180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 489.793892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 489.809301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 489.825084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 489.840193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 489.849384] LNet: 61207:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 489.857962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 489.872798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 489.886484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 489.898959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 489.912103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 489.925468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 489.940969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 489.955766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 490.162482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e remote_error -r 1 [ 490.177757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e remote_error -r 1 [ 490.193918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e remote_error -r 1 [ 490.210497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e remote_error -r 1 [ 490.226808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e remote_error -r 1 [ 490.242248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e remote_error -r 1 [ 490.261062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e remote_error -r 1 [ 490.277212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e remote_error -r 1 [ 490.293977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 490.303194] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 490.306304] LNet: Skipped 2 previous similar messages [ 490.394703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 490.408366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 490.422401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 490.436953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 490.451097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 490.463754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 490.477517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 490.490734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 490.498493] LNet: 61330:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 490.507358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 490.520904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 490.534347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 490.548809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 490.563850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 490.579997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 490.595378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 490.611028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 490.817640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e remote_timeout -r 1 [ 490.834973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e remote_timeout -r 1 [ 490.852213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e remote_timeout -r 1 [ 490.869017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e remote_timeout -r 1 [ 490.888887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e remote_timeout -r 1 [ 490.909239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e remote_timeout -r 1 [ 490.926601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e remote_timeout -r 1 [ 490.944525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e remote_timeout -r 1 [ 490.962607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 491.053706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 491.067180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 491.081618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 491.094901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 491.108957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 491.123918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 491.138066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 491.151379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 491.165152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 491.179221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 491.195601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 491.211891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 491.227136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 491.244281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 491.261036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 491.277944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 493.492168] LNet: 61531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.495445] LNetError: 61531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 493.499539] LNet: Removed LNI 192.168.202.46@tcp [ 493.518642] LNet: Removed LNI 192.168.202.46@tcp1 [ 493.666544] Key type .llcrypt unregistered [ 493.668036] Key type ._llcrypt unregistered [ 499.220388] Key type ._llcrypt registered [ 499.221517] Key type .llcrypt registered [ 499.751103] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 20:10:31 (1777421431) [ 499.945835] Key type .llcrypt unregistered [ 499.947062] Key type ._llcrypt unregistered [ 502.022601] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 502.091302] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 502.100928] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 502.102941] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 504.310060] Key type ._llcrypt registered [ 504.311437] Key type .llcrypt registered [ 504.371367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 504.378182] alg: No test for adler32 (adler32-zlib) [ 505.188836] LNet: 62606:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 505.195446] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 505.210232] LNet: 62620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 505.212469] LNetError: 62620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 505.222305] LNet: Removed LNI 192.168.202.46@tcp [ 505.475328] Key type .llcrypt unregistered [ 505.476692] Key type ._llcrypt unregistered [ 507.621374] Key type ._llcrypt registered [ 507.623097] Key type .llcrypt registered [ 507.708650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 507.717760] alg: No test for adler32 (adler32-zlib) [ 508.541042] LNet: 62879:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 508.550856] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 508.553644] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 508.569937] LNet: 62893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 508.573561] LNetError: 62893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 508.576712] LNet: Removed LNI 192.168.202.46@tcp [ 508.586430] LNet: Removed LNI 10.1.2.3@tcp1 [ 508.871403] Key type .llcrypt unregistered [ 508.873122] Key type ._llcrypt unregistered [ 511.059956] Key type ._llcrypt registered [ 511.061435] Key type .llcrypt registered [ 511.126084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 511.132981] alg: No test for adler32 (adler32-zlib) [ 511.941125] LNet: 63151:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 511.946518] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 511.948660] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 511.963807] LNet: 63165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 511.967524] LNetError: 63165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 511.970308] LNet: Removed LNI 192.168.202.46@tcp [ 511.983396] LNet: Removed LNI 10.1.2.3@tcp [ 512.231385] Key type .llcrypt unregistered [ 512.232695] Key type ._llcrypt unregistered [ 514.322534] Key type ._llcrypt registered [ 514.323806] Key type .llcrypt registered [ 514.383336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 514.389698] alg: No test for adler32 (adler32-zlib) [ 515.195777] LNet: 63425:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 515.201033] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 515.203167] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 515.217331] LNet: 63439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 515.220498] LNetError: 63439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 515.223398] LNet: Removed LNI 192.168.202.46@tcp [ 515.234546] LNet: Removed LNI 10.1.2.3@tcp [ 515.479677] Key type .llcrypt unregistered [ 515.480829] Key type ._llcrypt unregistered [ 517.601845] Key type ._llcrypt registered [ 517.604396] Key type .llcrypt registered [ 517.701206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 517.710833] alg: No test for adler32 (adler32-zlib) [ 518.530961] LNet: 63704:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 518.537656] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 518.539812] LNetError: 63704:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 518.542418] LNetError: Error -100 starting up LNI tcp [ 518.551957] LNet: Removed LNI 192.168.202.46@tcp [ 518.836292] Key type .llcrypt unregistered [ 518.837552] Key type ._llcrypt unregistered [ 522.746991] Key type ._llcrypt registered [ 522.748662] Key type .llcrypt registered [ 523.234306] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 20:10:54 (1777421454) [ 523.515490] Key type .llcrypt unregistered [ 523.517343] Key type ._llcrypt unregistered [ 525.483524] Key type ._llcrypt registered [ 525.484682] Key type .llcrypt registered [ 525.559400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 525.566396] alg: No test for adler32 (adler32-zlib) [ 526.373505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 526.390217] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 529.484786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 529.712682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e network_timeout -r 1 [ 529.730189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e network_timeout -r 1 [ 529.746432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 529.754841] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 529.836869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 529.850719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 529.863707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 529.877746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 529.891388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 529.906177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 529.921281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 529.934558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 529.943614] LNet: 64957:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 529.953083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 529.968027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 529.983764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 529.997037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 530.011975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 530.026401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 530.123133] LNet: 64996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 530.125994] LNetError: 64996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 530.139530] LNet: Removed LNI 192.168.202.46@tcp [ 530.289470] Key type .llcrypt unregistered [ 530.290804] Key type ._llcrypt unregistered [ 536.658116] Key type ._llcrypt registered [ 536.659650] Key type .llcrypt registered [ 536.816937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 536.825150] alg: No test for adler32 (adler32-zlib) [ 537.652981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 537.669033] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 540.921317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 543.952458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.968328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 543.979237] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 544.583757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e network_timeout -r 1 [ 544.604742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e network_timeout -r 1 [ 544.624881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e network_timeout -r 1 [ 544.643290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e network_timeout -r 1 [ 544.659708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e network_timeout -r 1 [ 544.674405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e network_timeout -r 1 [ 544.690275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e network_timeout -r 1 [ 544.708612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e network_timeout -r 1 [ 544.723664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 544.735949] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 544.820366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 544.833581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 544.848097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 544.862980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 544.865223] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 544.868047] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.146@tcp [ 544.875701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 544.887867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 544.904933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 544.919635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 544.928863] LNet: 65872:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 544.938422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 544.954254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 544.974587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 544.996501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 545.011253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 545.026652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 547.217614] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 547.333958] LNet: 65945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 547.337500] LNetError: 65945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 547.342484] LNet: Removed LNI 192.168.202.46@tcp [ 547.352458] LNet: Removed LNI 192.168.202.46@tcp1 [ 547.526587] Key type .llcrypt unregistered [ 547.528628] Key type ._llcrypt unregistered [ 553.613206] Key type ._llcrypt registered [ 553.614650] Key type .llcrypt registered [ 554.151381] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 20:11:25 (1777421485) [ 554.294794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 554.301303] alg: No test for adler32 (adler32-zlib) [ 555.120493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.149367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 555.162120] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 555.447806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 555.458397] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 555.817831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 555.844992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 555.866312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 555.884111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 555.903544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 555.915618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 555.932669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.46@tcp [ 555.944327] LNetError: 67044:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.202.46@tcp to recovery [ 556.192297] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 556.192339] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 558.240445] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 559.134858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 562.336312] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 563.352596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 563.370996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 563.401888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.414963] LNet: 67162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 563.418871] LNetError: 67162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 563.425700] LNet: Removed LNI 192.168.202.46@tcp [ 563.434633] LNet: Removed LNI 192.168.202.46@tcp1 [ 563.446523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.471182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 563.485762] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 563.832808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 563.844220] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 564.218107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 564.238086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 564.263707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 564.283936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 564.302363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 564.325927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 564.341898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 564.357875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.46@tcp [ 564.368113] LNetError: 67434:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.202.46@tcp to recovery [ 564.512362] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 567.573364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 570.656340] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 570.660963] LNet: Skipped 1 previous similar message [ 574.956794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 574.982672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 575.008163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 577.762532] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 20:11:49 (1777421509) [ 577.814511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.824277] LNet: 68146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 577.828184] LNetError: 68146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 577.830768] LNetError: 68146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 577.833970] LNet: Removed LNI 192.168.202.46@tcp [ 577.842762] LNet: Removed LNI 192.168.202.46@tcp1 [ 577.858016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.883287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 577.895221] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 578.211605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 578.220665] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 578.644791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 578.683766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 578.706081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 578.724498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 578.744829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 578.763121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 578.780207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.46@tcp --health 0 [ 578.824202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 578.912181] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.46@tcp [ 579.936264] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 579.939506] LNet: Skipped 1 previous similar message [ 593.372472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 593.453791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 593.468938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 593.486324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.46@tcp --health 500 [ 593.530028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 599.778817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 599.820177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.831153] LNet: 68642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 599.835400] LNetError: 68642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 599.838672] LNetError: 68642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 599.841745] LNet: Removed LNI 192.168.202.46@tcp [ 599.850406] LNet: Removed LNI 192.168.202.46@tcp1 [ 599.866680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.899312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 599.917813] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 600.260831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 600.275262] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 600.700796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 600.719062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 600.751262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 600.767237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 600.783098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 600.799482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 600.816552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 600.833124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.46@tcp --health 0 [ 601.952388] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 601.955314] LNet: Skipped 3 previous similar messages [ 602.990502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 612.456744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 612.474253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 612.490117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 615.404485] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 20:12:26 (1777421546) [ 615.602940] LNet: 69698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.607622] LNetError: 69698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.610380] LNetError: 69698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 615.614232] LNet: Removed LNI 192.168.202.46@tcp [ 615.624388] LNet: Removed LNI 192.168.202.46@tcp1 [ 615.926590] Key type .llcrypt unregistered [ 615.928238] Key type ._llcrypt unregistered [ 618.322234] Key type ._llcrypt registered [ 618.324080] Key type .llcrypt registered [ 618.423927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 618.434520] alg: No test for adler32 (adler32-zlib) [ 619.258744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 619.271656] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 623.482941] Lustre: DEBUG MARKER: Initial discovery [ 623.507019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.146@tcp [ 624.509226] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 624.547167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 624.564641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.146@tcp [ 624.582881] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 624.594769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 624.609250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.146@tcp -r 1 -m GET -l 3 [ 624.626758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.146@tcp -r 1 -m GET -e local_error [ 624.644900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.146@tcp -r 1 -m PUT -l 6 [ 625.266980] Lustre: DEBUG MARKER: Force oleg246-client.virtnet to discover 192.168.202.146@tcp (in background) [ 625.299548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.146@tcp [ 625.440277] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 625.893212] Lustre: DEBUG MARKER: Force oleg246-server to discover 192.168.202.46@tcp [ 626.848513] Lustre: DEBUG MARKER: Wait for 70452 [ 628.320324] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 628.840556] Lustre: DEBUG MARKER: Finished wait on 70452 [ 628.892819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 628.913530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 629.011513] LNet: 70907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.014637] LNetError: 70907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.021578] LNet: Removed LNI 192.168.202.46@tcp [ 629.184416] Key type .llcrypt unregistered [ 629.185954] Key type ._llcrypt unregistered [ 634.961326] Key type ._llcrypt registered [ 634.962319] Key type .llcrypt registered [ 635.576487] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 20:12:46 (1777421566) [ 635.788422] Key type .llcrypt unregistered [ 635.789947] Key type ._llcrypt unregistered [ 637.983328] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 638.030777] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 638.034038] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 638.037042] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 640.279284] Key type ._llcrypt registered [ 640.280546] Key type .llcrypt registered [ 640.381941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 640.390608] alg: No test for adler32 (adler32-zlib) [ 641.215220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.250562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 641.266424] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 641.623880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 641.635206] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 645.413195] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 20:12:56 (1777421576) [ 645.454286] LNet: 72969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.458206] LNetError: 72969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.464882] LNet: Removed LNI 192.168.202.46@tcp [ 645.472106] LNet: Removed LNI 10.1.2.3@tcp [ 645.808684] Key type .llcrypt unregistered [ 645.810272] Key type ._llcrypt unregistered [ 648.498372] Key type ._llcrypt registered [ 648.499730] Key type .llcrypt registered [ 648.576178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 648.584053] alg: No test for adler32 (adler32-zlib) [ 649.407249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.436795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 649.450162] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 649.803726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 649.813759] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 650.218945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.46@tcp [ 651.232100] LNetError: Unexpected error -22 connecting to 192.168.202.46@tcp at host 192.168.202.46:988 [ 652.768686] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 20:13:04 (1777421584) [ 652.804121] LNet: 74100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.807377] LNetError: 74100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 652.809809] LNet: Removed LNI 192.168.202.46@tcp [ 652.817495] LNet: Removed LNI 10.1.2.3@tcp [ 653.075463] Key type .llcrypt unregistered [ 653.076947] Key type ._llcrypt unregistered [ 655.370815] Key type ._llcrypt registered [ 655.373225] Key type .llcrypt registered [ 655.452785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 655.458953] alg: No test for adler32 (adler32-zlib) [ 656.279298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.305353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 656.319479] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 656.616460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 656.625787] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 656.972214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.46@tcp1 --nid 192.168.202.46@tcp2 [ 658.583311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp1 192.168.202.46@tcp2 [ 658.592937] LNetError: 74942:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.46@tcp1 to 192.168.202.46@tcp2 - Route table may be misconfigured [ 658.603767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp2 192.168.202.46@tcp1 [ 660.929130] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 20:13:12 (1777421592) [ 660.973118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.985481] LNet: 75510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.987698] LNetError: 75510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 660.990573] LNet: Removed LNI 192.168.202.46@tcp1 [ 662.054663] LNet: Removed LNI 192.168.202.46@tcp2 [ 662.066780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.095530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 662.105975] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 662.444353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 662.454816] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 662.832277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 662.853461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp 192.168.202.46@tcp [ 662.873265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp1 192.168.202.46@tcp1 [ 662.893155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.46@tcp -d 192.168.202.46@tcp -e network_timeout [ 662.910350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -e network_timeout [ 662.929568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -e network_timeout [ 662.947633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -e network_timeout [ 662.968034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 662.978779] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 662.989904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 663.040310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 663.084569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 663.136305] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 665.597698] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 20:13:16 (1777421596) [ 665.646214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.657105] LNet: 76374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.659961] LNetError: 76374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.663660] LNetError: 76374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 665.667453] LNet: Removed LNI 192.168.202.46@tcp [ 666.726804] LNet: Removed LNI 192.168.202.46@tcp1 [ 666.737336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.773900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 666.880624] LNet: 76414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.023340] Key type .llcrypt unregistered [ 667.024702] Key type ._llcrypt unregistered [ 670.959954] Key type ._llcrypt registered [ 670.961527] Key type .llcrypt registered [ 671.488630] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 20:13:22 (1777421602) [ 671.640370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 671.646172] alg: No test for adler32 (adler32-zlib) [ 672.456927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.997840] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 673.621142] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 20:13:25 (1777421605) [ 673.666400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.675695] LNet: 77507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.688476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.716292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 673.726568] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 674.018740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 674.028496] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 674.379226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 674.394876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp1 [ 674.413266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp1 [ 676.730550] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 20:13:28 (1777421608) [ 677.445108] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg246-server [ 678.220912] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 20:13:29 (1777421609) [ 678.974747] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg246-server [ 679.695389] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 20:13:31 (1777421611) [ 680.438309] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg246-server [ 681.199034] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 20:13:32 (1777421612) [ 682.067143] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg246-server [ 682.874650] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 20:13:34 (1777421614) [ 683.582184] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg246-server [ 684.267573] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 20:13:35 (1777421615) [ 684.955306] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg246-server [ 685.648401] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 20:13:37 (1777421617) [ 686.367781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg246-server [ 687.107604] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 20:13:38 (1777421618) [ 687.961131] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg246-server [ 688.786375] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 20:13:40 (1777421620) [ 689.578769] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg246-server [ 690.310725] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 20:13:41 (1777421621) [ 690.351521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.369263] LNet: 81379:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.376694] LNetError: 81379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 690.383829] LNet: Removed LNI 192.168.202.46@tcp [ 691.430329] LNet: Removed LNI 192.168.202.46@tcp1 [ 691.440743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.471321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 691.486073] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 691.815107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 691.850246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 691.916298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.926913] LNet: 81532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.934581] LNetError: 81532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.937511] LNetError: 81532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 691.948921] LNet: Removed LNI 192.168.202.46@tcp [ 691.959993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.985757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 691.999116] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 692.300333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 692.377265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.414530] LNet: Removed LNI 192.168.202.46@tcp [ 693.426984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.450560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 693.461641] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 693.747177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 693.781418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 694.901901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.910811] LNet: 81833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.913767] LNet: 81833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 694.916957] LNetError: 81833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 694.919886] LNetError: 81833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 694.927428] LNet: Removed LNI 192.168.202.46@tcp [ 694.936240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.960265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 694.971350] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 695.277130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 695.358779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.422441] LNet: Removed LNI 192.168.202.46@tcp [ 696.433711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.459259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 696.472359] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 696.822421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 696.859143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 696.941418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.951719] LNetError: 82126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.954184] LNetError: 82126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 696.966764] LNet: Removed LNI 192.168.202.46@tcp [ 696.976413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.001674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 697.013461] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 697.344163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 697.439627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.470418] LNet: Removed LNI 192.168.202.46@tcp [ 698.479824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.509917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 698.523496] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 698.865383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 698.897585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 698.986555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.996426] LNet: 82418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.999539] LNet: 82418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 699.013098] LNet: Removed LNI 192.168.202.46@tcp [ 699.022904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.047344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 699.062684] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 699.383397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 699.480358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.518567] LNet: Removed LNI 192.168.202.46@tcp [ 700.528510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.555472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 700.566394] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 700.920838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 700.956295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 702.117839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.130386] LNetError: 82720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 702.133849] LNetError: 82720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 702.139426] LNet: Removed LNI 192.168.202.46@tcp [ 702.152890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.182287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 702.195926] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 702.502224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 703.660258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.680217] LNet: Removed LNI 192.168.202.46@tcp [ 704.697078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.724608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 704.738551] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 705.074261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 705.117121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 705.226612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.277936] LNet: Removed LNI 192.168.202.46@tcp [ 706.289539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.314751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 706.326665] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 706.708559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 706.823434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.881514] LNet: Removed LNI 192.168.202.46@tcp [ 707.895310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.929293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 707.944419] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 708.277048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 708.302373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.311336] LNet: 83299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 708.314022] LNet: 83299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 708.327438] LNet: Removed LNI 192.168.202.46@tcp [ 708.338607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.366149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 708.377088] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 708.706789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 711.350261] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 20:14:02 (1777421642) [ 711.384939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.395575] LNetError: 83997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.398183] LNetError: 83997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 711.407592] LNet: Removed LNI 192.168.202.46@tcp [ 711.417799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.447265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 711.460456] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/180] [ 711.979972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.002370] LNet: Removed LNI 192.168.202.46@tcp231 [ 712.017716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.061137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 712.076754] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/180] [ 712.080095] LNet: 84148:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 712.245414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 712.263907] LNet: Removed LNI 192.168.202.46@tcp231 [ 712.271662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 712.283205] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/180] [ 712.797710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.820278] LNet: Removed LNI 192.168.202.46@tcp231 [ 712.832983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.868993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 712.881552] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/0] [ 712.886536] LNet: 84301:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 713.045160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 713.064478] LNet: Removed LNI 192.168.202.46@tcp231 [ 713.073680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 713.086203] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/0] [ 713.559886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.583540] LNet: Removed LNI 192.168.202.46@tcp231 [ 713.593346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.622544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 713.633936] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/60] [ 713.790247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 713.809807] LNet: Removed LNI 192.168.202.46@tcp231 [ 713.818608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 713.830317] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/60] [ 714.270588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.285438] LNet: Removed LNI 192.168.202.46@tcp231 [ 714.294287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.322809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 714.333334] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/180] [ 714.336036] LNet: 84603:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 714.339050] LNet: 84603:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 714.491177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 714.510362] LNet: Removed LNI 192.168.202.46@tcp231 [ 714.517642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 714.528706] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/180] [ 717.320383] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 20:14:08 (1777421648) [ 717.358153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.377478] LNet: Removed LNI 192.168.202.46@tcp231 [ 717.389340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.427987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 717.440770] LNet: Added LNI 192.168.202.46@tcp232 [8/256/0/180] [ 717.799494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 717.818149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 717.837912] LNet: Removed LNI 192.168.202.46@tcp232 [ 717.846596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 717.859670] LNet: Added LNI 192.168.202.46@tcp232 [8/256/0/180] [ 718.201602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 718.224912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 718.246768] LNet: Removed LNI 192.168.202.46@tcp232 [ 718.256875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 718.271028] LNet: Added LNI 192.168.202.46@tcp232 [8/256/0/180] [ 718.587168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 718.608305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 718.626921] LNet: Removed LNI 192.168.202.46@tcp232 [ 718.636303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 718.648865] LNet: Added LNI 192.168.202.46@tcp232 [8/256/0/180] [ 719.000931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 719.018130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 719.040509] LNet: Removed LNI 192.168.202.46@tcp232 [ 719.048219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 721.665821] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 20:14:12 (1777421652) [ 722.170334] Key type .llcrypt unregistered [ 722.174347] Key type ._llcrypt unregistered [ 724.586942] Key type ._llcrypt registered [ 724.588538] Key type .llcrypt registered [ 724.696883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 724.708803] alg: No test for adler32 (adler32-zlib) [ 725.555238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 725.577791] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 728.970219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 732.386948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.403842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 732.415569] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 733.015433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_interrupt -r 2 [ 733.037178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_interrupt -r 2 [ 733.055428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_interrupt -r 2 [ 733.074874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_interrupt -r 2 [ 733.094639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_interrupt -r 2 [ 733.112916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_interrupt -r 2 [ 733.132248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_interrupt -r 2 [ 733.154365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_interrupt -r 2 [ 733.173685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp 192.168.202.146@tcp [ 733.184586] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 733.188493] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 733.280399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 733.297966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 733.319326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 733.345094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 733.370435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 733.384426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 733.398554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 733.413047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 733.423737] LNet: 87099:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 733.432502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 733.446618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 733.461345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 733.476154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 733.490431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 733.506373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 733.701729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_dropped -r 2 [ 733.719201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_dropped -r 2 [ 733.735772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_dropped -r 2 [ 733.752834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_dropped -r 2 [ 733.771676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_dropped -r 2 [ 733.787796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_dropped -r 2 [ 733.804521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_dropped -r 2 [ 733.821665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_dropped -r 2 [ 733.837316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp 192.168.202.146@tcp [ 733.847473] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 733.928680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 733.943255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 733.957488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 733.971506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 733.986382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 733.999159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 734.012330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 734.025337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 734.034621] LNet: 87219:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 734.042076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 734.056702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 734.070814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 734.084166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 734.098173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 734.113132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 734.295616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_aborted -r 2 [ 734.310340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_aborted -r 2 [ 734.326455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_aborted -r 2 [ 734.341950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_aborted -r 2 [ 734.358739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_aborted -r 2 [ 734.376498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_aborted -r 2 [ 734.392058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_aborted -r 2 [ 734.408448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_aborted -r 2 [ 734.424157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp 192.168.202.146@tcp [ 734.518711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 734.532799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 734.548716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 734.563385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 734.579664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 734.595574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 734.609828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 734.622715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 734.634131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 734.646626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 734.660295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 734.672626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 734.687677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 734.707552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 734.911336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_no_route -r 2 [ 734.929494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_no_route -r 2 [ 734.949079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_no_route -r 2 [ 734.967813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_no_route -r 2 [ 734.985786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_no_route -r 2 [ 735.004761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_no_route -r 2 [ 735.024775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_no_route -r 2 [ 735.042594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_no_route -r 2 [ 735.059641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp 192.168.202.146@tcp [ 735.069816] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 735.072801] LNet: Skipped 2 previous similar messages [ 735.161798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 735.175419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 735.189757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 735.204737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 735.222751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 735.239899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 735.256303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 735.273283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 735.282674] LNet: 87459:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 735.285731] LNet: 87459:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 735.295386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 735.312165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 735.335945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 735.352932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 735.371263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 735.387799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 735.597969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_timeout -r 2 [ 735.614219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_timeout -r 2 [ 735.631767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_timeout -r 2 [ 735.647850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_timeout -r 2 [ 735.665513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_timeout -r 2 [ 735.682252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_timeout -r 2 [ 735.699298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_timeout -r 2 [ 735.715556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_timeout -r 2 [ 735.732892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp 192.168.202.146@tcp [ 735.831833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 735.845596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 735.860290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 735.874368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 735.890051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 735.905484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 735.920479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 735.934499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 735.948135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 735.962351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 735.976909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 735.991800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 736.007726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 736.023607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 736.219515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e remote_dropped -r 2 [ 736.236910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e remote_dropped -r 2 [ 736.254940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e remote_dropped -r 2 [ 736.274078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e remote_dropped -r 2 [ 736.291529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e remote_dropped -r 2 [ 736.309342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e remote_dropped -r 2 [ 736.328117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e remote_dropped -r 2 [ 736.345561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e remote_dropped -r 2 [ 736.363868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp 192.168.202.146@tcp [ 736.374906] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.146@tcp [ 736.464990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 736.480942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 736.495978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 736.512504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 736.527905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 736.545614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 736.560814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 736.577142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 736.593533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 736.609543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 736.627692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 736.643716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 736.658709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 736.673441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 738.873559] LNet: 87773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.876489] LNetError: 87773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 738.880191] LNet: Removed LNI 192.168.202.46@tcp [ 738.888330] LNet: Removed LNI 192.168.202.46@tcp1 [ 739.051744] Key type .llcrypt unregistered [ 739.053077] Key type ._llcrypt unregistered [ 744.575391] Key type ._llcrypt registered [ 744.576595] Key type .llcrypt registered [ 745.088976] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 20:14:36 (1777421676) [ 745.224551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 745.230368] alg: No test for adler32 (adler32-zlib) [ 746.041675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.066798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 746.078506] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 746.407467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 746.422528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.46@tcp [ 746.431852] LNetError: 88749:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.202.46@tcp to recovery [ 746.466727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.46@tcp [ 746.504959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 748.862966] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 20:14:40 (1777421680) [ 748.894430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.904892] LNet: 89332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.908300] LNetError: 89332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 748.916422] LNet: Removed LNI 192.168.202.46@tcp [ 748.928322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.955398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 748.968129] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 749.266698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 749.276596] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 749.632212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.46@tcp [ 749.641534] LNetError: 89604:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.202.46@tcp to recovery [ 749.984288] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 749.984907] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 750.739628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.46@tcp [ 754.085015] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 20:14:45 (1777421685) [ 754.795779] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg246-server [ 755.488309] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 20:14:46 (1777421686) [ 756.148851] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg246-server [ 756.793347] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 20:14:48 (1777421688) [ 757.504157] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg246-server [ 758.176170] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 20:14:49 (1777421689) [ 758.225072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.236102] LNet: 91222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.239775] LNetError: 91222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.242959] LNet: Removed LNI 192.168.202.46@tcp [ 758.250329] LNet: Removed LNI 192.168.202.46@tcp1 [ 758.261057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.285827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 758.297564] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 758.593787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 758.626337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 758.648543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 758.680674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 758.699301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 758.715721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 761.248417] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 20:14:52 (1777421692) [ 761.289842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.301508] LNet: 91944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.304974] LNetError: 91944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 761.307964] LNetError: 91944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 761.318787] LNet: Removed LNI 192.168.202.46@tcp [ 761.341099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.367128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 761.379374] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 764.169841] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 20:14:55 (1777421695) [ 764.203235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 764.213288] LNet: 92646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.216636] LNetError: 92646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 764.223666] LNet: Removed LNI 192.168.202.46@tcp [ 764.235021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.266631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 764.276735] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 767.196181] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 20:14:58 (1777421698) [ 767.829899] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 768.537226] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 20:14:59 (1777421699) [ 768.716468] LNet: 93686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.722485] LNetError: 93686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.735392] LNet: Removed LNI 192.168.202.46@tcp [ 768.976674] Key type .llcrypt unregistered [ 768.978058] Key type ._llcrypt unregistered [ 771.122054] Key type ._llcrypt registered [ 771.123722] Key type .llcrypt registered [ 771.209866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 771.216353] alg: No test for adler32 (adler32-zlib) [ 772.046924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 772.065118] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 775.378872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 777.301892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.146@tcp [ 777.478650] LNet: 94101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.483944] LNetError: 94101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.491759] LNet: Removed LNI 192.168.202.46@tcp [ 777.661623] Key type .llcrypt unregistered [ 777.663090] Key type ._llcrypt unregistered [ 782.382585] Key type ._llcrypt registered [ 782.384058] Key type .llcrypt registered [ 782.995489] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 783.719824] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 784.452709] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 20:15:15 (1777421715) [ 784.587230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 784.593173] alg: No test for adler32 (adler32-zlib) [ 785.409492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.447194] LNet: 95186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.699544] Key type .llcrypt unregistered [ 785.701175] Key type ._llcrypt unregistered [ 787.976796] Key type ._llcrypt registered [ 787.978316] Key type .llcrypt registered [ 788.041838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 788.048755] alg: No test for adler32 (adler32-zlib) [ 788.880087] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 788.881558] LNet: 95441:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 788.886616] LNet: 95441:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 788.902435] LNetError: 95453:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.48@tcp is being used as a gateway but routing feature is not turned on [ 788.908389] LNetError: 95453:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.202.48@tcp has gone from up to down [ 789.102577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.224088] LNetError: 95453:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.202.49@tcp has gone from up to down [ 792.227744] LNetError: 95453:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 792.230806] LNet: 95500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.233647] LNetError: 95500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.242362] LNet: Removed LNI 192.168.202.46@tcp [ 792.252906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.283377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 792.295819] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 792.299023] LNet: 95507:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 792.303593] LNet: 95507:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 792.307778] LNetError: 95507:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.202.48@tcp is a gateway. Can not delete it [ 792.310781] LNetError: 95507:0:(peer.c:2159:lnet_del_peer_ni()) 192.168.202.48@tcp is a router. Can not be deleted [ 792.315871] LNet: 95507:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 792.319310] LNetError: 95504:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.50@tcp is being used as a gateway but routing feature is not turned on [ 792.326874] LNetError: 95504:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 794.907656] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 20:15:26 (1777421726) [ 795.040327] LNetError: 95510:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.202.51@tcp has gone from up to down [ 795.049397] LNetError: 95510:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 795.612770] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg246-server [ 796.295273] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 20:15:27 (1777421727) [ 797.056068] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg246-server [ 797.817281] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 20:15:29 (1777421729) [ 798.535511] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 800.248129] Key type lgssc registered [ 800.689714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 809.586800] Key type lgssc unregistered [ 809.744349] LNet: 97924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.747878] LNetError: 97924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.758142] LNet: Removed LNI 192.168.202.46@tcp [ 810.028660] Key type .llcrypt unregistered [ 810.030056] Key type ._llcrypt unregistered [ 815.689102] Key type ._llcrypt registered [ 815.690556] Key type .llcrypt registered [ 816.235429] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 20:15:47 (1777421747) [ 816.382743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 816.389289] alg: No test for adler32 (adler32-zlib) [ 817.203284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.231526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 817.251230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 817.273610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 817.294414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 819.851786] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 20:15:51 (1777421751) [ 819.915233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.927334] LNet: 99328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.941117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.967978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 819.981847] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 820.392564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.401903] LNetError: 99472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.409539] LNet: Removed LNI 192.168.202.46@tcp [ 820.417629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.441277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 820.453397] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 820.634358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.645547] LNet: 99500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.648432] LNet: 99500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 820.655488] LNet: Removed LNI 192.168.202.46@tcp [ 820.664839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.687358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 820.700520] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 823.338841] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 20:15:54 (1777421754) [ 823.395665] LNet: 100088:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.398971] LNetError: 100088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.402113] LNetError: 100088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 823.410759] LNet: Removed LNI 192.168.202.46@tcp [ 823.667642] Key type .llcrypt unregistered [ 823.669361] Key type ._llcrypt unregistered [ 825.986596] Key type ._llcrypt registered [ 825.988477] Key type .llcrypt registered [ 826.064342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 826.069835] alg: No test for adler32 (adler32-zlib) [ 826.895109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.936907] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 826.939930] LNet: *** cfs_fail_loc=e003, val=100*** [ 827.270743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 827.283291] Ping reply received with mapping: [ 827.285276] found metadata reply NI[192.168.202.46@tcp] [ 827.287046] 2e ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 827.289219] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 827.291199] 5a 5a 5a 5a 5a 5a 5a 5a [ 827.342571] LNet: 100487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.345779] LNetError: 100487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.356340] LNet: Removed LNI 192.168.202.46@tcp [ 827.605587] Key type .llcrypt unregistered [ 827.607371] Key type ._llcrypt unregistered [ 831.797516] Key type ._llcrypt registered [ 831.798891] Key type .llcrypt registered [ 832.412818] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 20:16:03 (1777421763) [ 832.607874] Key type .llcrypt unregistered [ 832.609143] Key type ._llcrypt unregistered [ 834.903972] Key type ._llcrypt registered [ 834.905379] Key type .llcrypt registered [ 834.980451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 834.988822] alg: No test for adler32 (adler32-zlib) [ 835.881114] LNetError: 101539:0:(api-ni.c:2734:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 836.036467] Key type .llcrypt unregistered [ 836.038882] Key type ._llcrypt unregistered [ 838.039221] Key type ._llcrypt registered [ 838.040751] Key type .llcrypt registered [ 838.560542] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 20:16:09 (1777421769) [ 838.770064] Key type .llcrypt unregistered [ 838.771591] Key type ._llcrypt unregistered [ 840.900067] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 840.902708] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 843.101042] Key type ._llcrypt registered [ 843.102373] Key type .llcrypt registered [ 843.165756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 843.171589] alg: No test for adler32 (adler32-zlib) [ 843.981857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.010254] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 848.262472] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 20:16:19 (1777421779) [ 848.293531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 848.306699] LNet: 103311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.310206] LNetError: 103311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 848.319829] LNet: Removed LNI 10.1.2.3@tcp [ 848.329639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.613584] Key type .llcrypt unregistered [ 848.615564] Key type ._llcrypt unregistered [ 851.050579] Key type ._llcrypt registered [ 851.051972] Key type .llcrypt registered [ 851.141715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 851.150408] alg: No test for adler32 (adler32-zlib) [ 851.976965] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 852.002935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 852.073552] LNet: 103604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.077969] LNetError: 103604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.087790] LNet: Removed LNI 192.168.202.46@tcp [ 852.348171] Key type .llcrypt unregistered [ 852.349869] Key type ._llcrypt unregistered [ 856.735304] Key type ._llcrypt registered [ 856.736851] Key type .llcrypt registered [ 857.253153] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 20:16:28 (1777421788) [ 857.486484] Key type .llcrypt unregistered [ 857.488129] Key type ._llcrypt unregistered [ 859.903704] Key type ._llcrypt registered [ 859.904841] Key type .llcrypt registered [ 859.974707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 859.980420] alg: No test for adler32 (adler32-zlib) [ 863.898835] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 20:16:35 (1777421795) [ 864.015196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.050352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.082962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 864.098752] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 864.487505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 864.508681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 864.527793] LNet: Removed LNI 192.168.202.46@tcp [ 864.686840] LNet: 105345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.949415] Key type .llcrypt unregistered [ 864.951766] Key type ._llcrypt unregistered [ 871.783038] Key type ._llcrypt registered [ 871.786795] Key type .llcrypt registered [ 872.601636] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 20:16:43 (1777421803) [ 872.918625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 872.929782] alg: No test for adler32 (adler32-zlib) [ 873.770661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.812955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 873.828657] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 874.341792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 874.370743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.46@tcp [ 874.383823] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 874.386415] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 874.388837] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 874, 0, 0 [ 874.392107] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 874.393953] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 874.396569] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 874.398925] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 874, 0, 0 [ 874.402680] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 874.404629] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 874.410352] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 874.412729] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 874, 0, 924 [ 874.416636] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 874.418988] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 874.421843] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 874.424473] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 874, 0, 924 [ 874.427939] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 878.691954] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 20:16:49 (1777421809) [ 879.014963] LNet: 106885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.018853] LNetError: 106885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.029893] LNet: Removed LNI 192.168.202.46@tcp [ 879.435373] Key type .llcrypt unregistered [ 879.439209] Key type ._llcrypt unregistered [ 883.050827] Key type ._llcrypt registered [ 883.052590] Key type .llcrypt registered [ 883.173870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 883.184957] alg: No test for adler32 (adler32-zlib) [ 884.056507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 884.078955] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 888.343339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 892.659379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.677695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 892.690562] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 893.156939] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 893.204805] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 893.209539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 893.215953] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 895.393960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 895.404363] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 900.830190] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 20:17:12 (1777421832) [ 900.879093] LNet: 108377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.883190] LNetError: 108377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.887762] LNet: Removed LNI 192.168.202.46@tcp [ 901.920144] LNet: Removed LNI 192.168.202.46@tcp1 [ 901.924846] LNetError: 108377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 901.929553] LNetError: 108377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 901.942581] LNet: Removed LNI 10.1.2.3@tcp99 [ 902.270503] Key type .llcrypt unregistered [ 902.273519] Key type ._llcrypt unregistered [ 904.778878] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 904.783262] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 907.030402] Key type ._llcrypt registered [ 907.032255] Key type .llcrypt registered [ 907.121656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 907.129859] alg: No test for adler32 (adler32-zlib) [ 907.953729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.985118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 908.002317] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 908.373619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 908.383269] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 909.347088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 909.364677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 909.383968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 909.392602] LNetError: 109129:0:(peer.c:2167:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 909.406729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 909.425187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 909.590705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 909.626635] LNet: 109149:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.630559] LNetError: 109149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.633481] LNet: Removed LNI 192.168.202.46@tcp [ 909.640284] LNet: Removed LNI 10.1.2.3@tcp [ 909.895411] Key type .llcrypt unregistered [ 909.897154] Key type ._llcrypt unregistered [ 913.968777] Key type ._llcrypt registered [ 913.970404] Key type .llcrypt registered [ 914.550365] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 20:17:25 (1777421845) [ 914.680128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 914.686966] alg: No test for adler32 (adler32-zlib) [ 915.499919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.523380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 915.536379] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 915.826428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 915.862353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg246-client@tcp [ 918.310514] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 20:17:29 (1777421849) [ 918.369146] LNet: 110675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.372362] LNetError: 110675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.380125] LNet: Removed LNI 192.168.202.46@tcp [ 918.663417] Key type .llcrypt unregistered [ 918.664668] Key type ._llcrypt unregistered [ 920.794838] Key type ._llcrypt registered [ 920.795772] Key type .llcrypt registered [ 920.860776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 920.867877] alg: No test for adler32 (adler32-zlib) [ 921.675269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 921.687812] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 921.700903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 924.132114] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 20:17:35 (1777421855) [ 924.760977] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 925.407542] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 20:17:36 (1777421856) [ 925.981932] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 926.670167] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 20:17:38 (1777421858) [ 926.710846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.721164] LNet: 112094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.724720] LNetError: 112094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.734478] LNet: Removed LNI 192.168.202.46@tcp [ 926.745267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.770796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 926.783233] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 927.093573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 927.115613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 930.040695] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 20:17:41 (1777421861) [ 930.127996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.145416] LNet: 112807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.149677] LNetError: 112807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.164058] LNet: Removed LNI 192.168.202.46@tcp [ 930.172404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.202171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 930.217507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 930.508307] Key type .llcrypt unregistered [ 930.509873] Key type ._llcrypt unregistered [ 935.534642] Key type ._llcrypt registered [ 935.536100] Key type .llcrypt registered [ 936.200681] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 20:17:47 (1777421867) [ 936.355915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 936.364064] alg: No test for adler32 (adler32-zlib) [ 937.214696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.254944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 937.270134] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 937.644257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 937.679656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 940.348397] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 20:17:51 (1777421871) [ 940.397710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 940.407497] LNet: 114358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.411826] LNetError: 114358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.425419] LNet: Removed LNI 192.168.202.46@tcp [ 940.436561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.461350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 940.477127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 943.235786] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 20:17:54 (1777421874) [ 943.276740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.288094] LNet: 114934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.301885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.358276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 943.540254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 943.561457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 943.599840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 943.794568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 943.816530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 943.851873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.872697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.911286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 944.105982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 944.125311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 944.152987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 944.330502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 944.351051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 947.210897] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 20:17:58 (1777421878) [ 947.272342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.284305] LNet: 115572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.289309] LNet: 115572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 947.314791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.349896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 947.368519] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 947.738209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 947.759024] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 948.255410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 948.286781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 948.409300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.421676] LNetError: 115852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.424576] LNet: Removed LNI 192.168.202.46@tcp [ 948.435815] LNet: Removed LNI 192.168.202.46@tcp2 [ 948.451736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.496510] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 948.506811] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 948.509333] LNet: 115858:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 948.749804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 948.855191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.920810] LNet: Removed LNI 192.168.202.46@tcp [ 949.922446] LNetError: 115895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.925441] LNetError: 115895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 949.937400] LNet: Removed LNI 192.168.202.46@tcp2 [ 949.949198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.986833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 950.002581] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 950.439091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 950.455657] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 950.883264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 950.903094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 950.939196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 951.025309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.035384] LNet: 116179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.037900] LNet: 116179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 952.098158] LNet: Removed LNI 192.168.202.46@tcp [ 952.100143] LNetError: 116179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.106216] LNetError: 116179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 952.117629] LNet: Removed LNI 192.168.202.46@tcp2 [ 952.131917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.163095] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 952.171986] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 952.176628] LNet: 116185:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 952.406547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 952.525978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.544680] LNet: Removed LNI 192.168.202.46@tcp [ 953.575123] LNet: Removed LNI 192.168.202.46@tcp2 [ 953.586765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.617939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 953.631523] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 954.005495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 954.020293] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 954.467882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 954.507186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 954.662392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 955.680439] LNet: Removed LNI 192.168.202.46@tcp [ 955.688882] LNet: Removed LNI 192.168.202.46@tcp2 [ 955.705203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.749088] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 955.754257] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 955.758257] LNet: 116508:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 956.032475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 956.130355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.142771] LNet: 116545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.145730] LNet: 116545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 956.149884] LNetError: 116545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.154573] LNetError: 116545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 956.158589] LNet: Removed LNI 192.168.202.46@tcp [ 956.169383] LNet: Removed LNI 192.168.202.46@tcp2 [ 956.183396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.217976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 956.236451] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 956.702974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 956.729326] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 957.121837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 957.141770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 957.176318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 957.277748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.306192] LNet: Removed LNI 192.168.202.46@tcp [ 959.333699] LNet: Removed LNI 192.168.202.46@tcp2 [ 959.351320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.387833] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 959.392169] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 959.396291] LNet: 116835:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 959.655333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 959.768419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 959.781517] LNet: Removed LNI 192.168.202.46@tcp [ 960.810159] LNet: Removed LNI 192.168.202.46@tcp2 [ 960.837277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.866473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 960.880414] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 961.252134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 961.266289] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 961.704280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 961.730446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 961.759567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 961.913413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.977339] LNet: Removed LNI 192.168.202.46@tcp [ 962.985405] LNet: Removed LNI 192.168.202.46@tcp2 [ 962.996687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.030159] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 963.034793] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 963.259624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 963.474449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 964.512167] LNet: Removed LNI 192.168.202.46@tcp [ 964.514465] LNetError: 117227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.518694] LNetError: 117227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 964.529776] LNet: Removed LNI 192.168.202.46@tcp2 [ 964.553358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.605694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 964.623579] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 965.004965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 965.018143] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 965.415775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 965.436819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 965.452716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 965.480695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 965.642830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.652662] LNet: 117528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.655022] LNet: 117528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 966.688317] LNet: Removed LNI 192.168.202.46@tcp [ 967.721756] LNet: Removed LNI 192.168.202.46@tcp2 [ 967.736178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.773175] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 967.776758] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 967.780081] LNet: 117535:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 967.783215] LNet: 117535:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 967.992486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 971.416866] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 972.129248] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 20:18:23 (1777421903) [ 972.172942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 972.183850] LNet: Removed LNI 192.168.202.46@tcp [ 973.226974] LNet: Removed LNI 192.168.202.46@tcp2 [ 973.240810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.371315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 973.383674] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 973.711646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 973.722196] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 974.124355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.134840] LNet: Removed LNI 192.168.202.46@tcp [ 974.147288] LNet: Removed LNI 10.1.2.3@tcp [ 974.156803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.184603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 974.199744] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 974.202529] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 974.364141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 974.374513] LNet: Removed LNI 192.168.202.46@tcp [ 974.382564] LNet: Removed LNI 10.1.2.3@tcp [ 976.841095] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 20:18:28 (1777421908) [ 976.906513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 976.925516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 976.970105] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 977.023459] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 977.025789] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 977.027902] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 979.213350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 979.229640] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 979.570518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 979.580293] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 980.438831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 980.458257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 980.485354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.46@tcp -m PUT -e local_timeout -r 1 [ 980.517208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 980.544305] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 980.549663] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 980.658621] LNetError: 119230:0:(lib-move.c:4960:lnet_parse()) 192.168.202.46@tcp, src 192.168.202.46@tcp: Dropping PUT (error -108 looking up sender) [ 981.664425] LNet: 119688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.667839] LNet: 119688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 981.671565] LNetError: 119688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.674224] LNetError: 119688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 981.680717] LNet: Removed LNI 192.168.202.46@tcp [ 981.686643] LNet: Removed LNI 10.1.2.3@tcp [ 981.880748] Key type .llcrypt unregistered [ 981.882559] Key type ._llcrypt unregistered [ 986.859346] Key type ._llcrypt registered [ 986.860489] Key type .llcrypt registered [ 987.436580] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 20:18:38 (1777421918) [ 987.608750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 987.615653] alg: No test for adler32 (adler32-zlib) [ 988.475583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.689464] LNet: 120540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.938743] Key type .llcrypt unregistered [ 988.940530] Key type ._llcrypt unregistered [ 991.192359] Key type ._llcrypt registered [ 991.193540] Key type .llcrypt registered [ 991.265133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 991.273921] alg: No test for adler32 (adler32-zlib) [ 992.086179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 992.100602] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 995.252918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 995.324731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 995.569935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.146@tcp [ 995.674687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 995.692103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 995.707751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 995.722560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 995.737982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 995.754710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 995.769525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 995.783894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 995.792989] LNet: 120988:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 995.801660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 995.817435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 995.834739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 995.854290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 995.870259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 995.887716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 996.224696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 996.335871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 996.351054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 996.373067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 996.392043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 996.412359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 996.426678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 996.459714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 996.481648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 996.490711] LNet: 121105:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 996.507401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 996.519915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 996.537047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 996.562993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 996.583551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 996.599108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 996.909832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.146@tcp [ 997.048485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 997.068749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 997.089677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 997.106630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 997.122199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 997.137699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 997.157117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 997.172669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 997.188791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 997.209440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 997.219201] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.146@tcp [ 997.229195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 997.245306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 997.265930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 997.284814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 997.604930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 997.724912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 997.738152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 997.753431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 997.767350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 997.782206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 997.797957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 997.818584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 997.836458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 997.845813] LNet: 121339:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 997.849498] LNet: 121339:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 997.860751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 997.878466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 997.896082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 997.911980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 997.929996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 997.949942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 998.105707] LNet: 121385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.110819] LNetError: 121385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.122918] LNet: Removed LNI 192.168.202.46@tcp [ 998.290773] Key type .llcrypt unregistered [ 998.292602] Key type ._llcrypt unregistered [ 1005.304044] Key type ._llcrypt registered [ 1005.305590] Key type .llcrypt registered [ 1005.868551] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 20:18:57 (1777421937) [ 1006.047955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1006.057398] alg: No test for adler32 (adler32-zlib) [ 1006.875921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.044125] LNet: 122270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.317543] Key type .llcrypt unregistered [ 1007.319030] Key type ._llcrypt unregistered [ 1009.696587] Key type ._llcrypt registered [ 1009.698125] Key type .llcrypt registered [ 1009.787329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.795532] alg: No test for adler32 (adler32-zlib) [ 1010.619025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1010.631531] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1013.819915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 1017.150477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1017.168671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1017.181057] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 1017.563542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1017.807019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.146@tcp [ 1017.834955] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.146@tcp [ 1017.921288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1017.936528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1017.958616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1017.973524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1017.987271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1018.001660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1018.017153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1018.035192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1018.047516] LNet: 122920:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1018.057605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1018.073232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1018.093843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1018.119404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1018.141836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1018.159059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1018.422432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1018.531689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1018.551502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1018.567707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1018.581288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1018.596099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1018.610138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1018.624616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1018.638725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1018.646825] LNet: 123037:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1018.656129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1018.670644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1018.694602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1018.717589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1018.735963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1018.760064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1019.030158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.146@tcp [ 1019.137160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1019.157418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1019.173449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1019.189951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1019.207496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1019.231886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1019.252042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1019.273794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1019.298692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1019.317803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1019.334983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1019.352400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1019.368407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1019.388371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1019.644267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1019.737202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1019.754421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1019.769067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1019.784797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1019.800606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1019.815207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1019.829374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1019.843318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1019.851665] LNet: 123271:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1019.854667] LNet: 123271:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 1019.863086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1019.880186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1019.898354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1019.912035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1019.928126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1019.945466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1022.403336] LNet: 123352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.406577] LNetError: 123352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.410702] LNet: Removed LNI 192.168.202.46@tcp [ 1022.420601] LNet: Removed LNI 192.168.202.46@tcp1 [ 1022.603760] Key type .llcrypt unregistered [ 1022.605427] Key type ._llcrypt unregistered [ 1029.527646] Key type ._llcrypt registered [ 1029.529013] Key type .llcrypt registered [ 1030.267707] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 20:19:21 (1777421961) [ 1033.398618] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 20:19:24 (1777421964) [ 1036.426454] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 20:19:27 (1777421967) [ 1039.353121] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 20:19:30 (1777421970) [ 1042.319997] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 20:19:33 (1777421973) [ 1045.334587] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 20:19:36 (1777421976) [ 1048.490404] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 20:19:39 (1777421979) [ 1051.752919] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 20:19:43 (1777421983) [ 1055.219271] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 20:19:46 (1777421986) [ 1058.179489] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 20:19:49 (1777421989) [ 1061.038409] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 20:19:52 (1777421992) [ 1064.013147] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 20:19:55 (1777421995) [ 1066.854778] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 20:19:58 (1777421998) [ 1069.557269] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 20:20:01 (1777422001) [ 1071.975983] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 20:20:03 (1777422003) [ 1074.438527] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 20:20:05 (1777422005) [ 1077.088532] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 20:20:08 (1777422008) [ 1079.756132] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 20:20:11 (1777422011) [ 1079.890885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1079.898433] alg: No test for adler32 (adler32-zlib) [ 1080.709707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1080.767338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1080.782584] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1081.269617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1081.279656] LNet: 134049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.282891] LNetError: 134049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.292851] LNet: Removed LNI 192.168.202.46@tcp [ 1081.306359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.333601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1081.348135] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1084.123572] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 20:20:15 (1777422015) [ 1086.978479] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 20:20:18 (1777422018) [ 1089.790242] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 20:20:21 (1777422021) [ 1092.617255] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 20:20:24 (1777422024) [ 1095.127755] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 20:20:26 (1777422026) [ 1097.992853] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 20:20:29 (1777422029) [ 1100.868713] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 20:20:32 (1777422032) [ 1103.811179] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 20:20:35 (1777422035) [ 1106.643590] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 20:20:38 (1777422038) [ 1109.261646] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 20:20:40 (1777422040) [ 1111.671314] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 20:20:43 (1777422043) [ 1114.311806] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 20:20:45 (1777422045) [ 1117.166465] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 987 sec =========== 20:20:48 (1777422048) [ 1117.291810] LNet: 141505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1117.295584] LNetError: 141505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1117.312158] LNet: Removed LNI 192.168.202.46@tcp [ 1117.601571] Key type .llcrypt unregistered [ 1117.603272] Key type ._llcrypt unregistered [ 1122.273410] Key type ._llcrypt registered [ 1122.278060] Key type .llcrypt registered [ 1122.451135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1122.458375] alg: No test for adler32 (adler32-zlib) [ 1123.331519] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 1123.424369] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1125.008237] Key type lgssc registered [ 1125.477345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1149.134892] Lustre: Mounted lustre-client [ 1151.494808] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1158.823071] LustreError: 143952:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff92b6888d7000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1158.843906] Lustre: Unmounted lustre-client [ 1170.799902] Key type lgssc unregistered [ 1170.945528] LNet: 144432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1170.949678] LNetError: 144432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.006815] LNet: Removed LNI 192.168.202.46@tcp [ 1172.250118] Key type .llcrypt unregistered [ 1172.251306] Key type ._llcrypt unregistered