[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 607983231 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002647] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004033] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007055] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008032] pid_max: default: 32768 minimum: 301 [ 0.010110] LSM: Security Framework initializing [ 0.011111] Yama: becoming mindful. [ 0.012111] SELinux: Initializing. [ 0.014167] *** VALIDATE selinux *** [ 0.033085] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.040690] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.041311] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.043094] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.045109] *** VALIDATE tmpfs *** [ 0.047339] *** VALIDATE proc *** [ 0.048414] *** VALIDATE cgroup *** [ 0.049023] *** VALIDATE cgroup2 *** [ 0.051003] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.053077] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.054023] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.055078] Spectre V2 : User space: Vulnerable [ 0.057028] Speculative Store Bypass: Vulnerable [ 0.063425] debug: unmapping init [mem 0xffffffffb5059000-0xffffffffb5060fff] [ 0.067922] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.070377] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.071060] ... version: 2 [ 0.072028] ... bit width: 48 [ 0.073038] ... generic registers: 4 [ 0.074037] ... value mask: 0000ffffffffffff [ 0.075122] ... max period: 00007fffffffffff [ 0.076029] ... fixed-purpose events: 3 [ 0.077018] ... event mask: 000000070000000f [ 0.078419] rcu: Hierarchical SRCU implementation. [ 0.084400] smp: Bringing up secondary CPUs ... [ 0.087058] x86: Booting SMP configuration: [ 0.088037] .... node #0, CPUs: #1 #2 #3 [ 0.101158] smp: Brought up 1 node, 4 CPUs [ 0.103029] smpboot: Max logical packages: 1 [ 0.104020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.207012] node 0 deferred pages initialised in 101ms [ 0.211006] devtmpfs: initialized [ 0.212324] x86/mm: Memory block size: 128MB [ 0.217076] gcov: version magic: 0x41383552 [ 0.219582] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.225131] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.228509] pinctrl core: initialized pinctrl subsystem [ 0.232419] [ 0.233013] ************************************************************* [ 0.236027] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.238055] ** ** [ 0.241012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.244014] ** ** [ 0.246009] ** This means that this kernel is built to expose internal ** [ 0.249013] ** IOMMU data structures, which may compromise security on ** [ 0.251025] ** your system. ** [ 0.255063] ** ** [ 0.259014] ** If you see this message and you are not debugging the ** [ 0.262028] ** kernel, report this immediately to your vendor! ** [ 0.265023] ** ** [ 0.267019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.269010] ************************************************************* [ 0.274394] NET: Registered protocol family 16 [ 0.276579] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.279214] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.283090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.287212] cpuidle: using governor menu [ 0.290385] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.294722] PCI: Using configuration type 1 for base access [ 0.296422] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.306039] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.308016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.312182] cryptd: max_cpu_qlen set to 1000 [ 0.313415] ACPI: Added _OSI(Module Device) [ 0.314000] ACPI: Added _OSI(Processor Device) [ 0.314024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.316009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.323076] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.329432] ACPI: Interpreter enabled [ 0.330084] ACPI: PM: (supports S0 S3 S4 S5) [ 0.332026] ACPI: Using IOAPIC for interrupt routing [ 0.335212] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.339000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.353284] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.355031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.360018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.364287] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.373111] acpiphp: Slot [2] registered [ 0.374097] acpiphp: Slot [5] registered [ 0.375154] acpiphp: Slot [6] registered [ 0.376080] acpiphp: Slot [3] registered [ 0.378141] acpiphp: Slot [4] registered [ 0.380179] acpiphp: Slot [7] registered [ 0.381129] acpiphp: Slot [8] registered [ 0.383110] acpiphp: Slot [9] registered [ 0.385137] acpiphp: Slot [10] registered [ 0.387287] acpiphp: Slot [11] registered [ 0.388205] acpiphp: Slot [12] registered [ 0.389098] acpiphp: Slot [13] registered [ 0.390000] acpiphp: Slot [14] registered [ 0.390000] acpiphp: Slot [15] registered [ 0.393099] acpiphp: Slot [16] registered [ 0.395099] acpiphp: Slot [17] registered [ 0.397155] acpiphp: Slot [18] registered [ 0.399132] acpiphp: Slot [19] registered [ 0.400088] acpiphp: Slot [20] registered [ 0.402113] acpiphp: Slot [21] registered [ 0.404174] acpiphp: Slot [22] registered [ 0.406126] acpiphp: Slot [23] registered [ 0.408125] acpiphp: Slot [24] registered [ 0.411107] acpiphp: Slot [25] registered [ 0.413215] acpiphp: Slot [26] registered [ 0.415113] acpiphp: Slot [27] registered [ 0.417129] acpiphp: Slot [28] registered [ 0.419233] acpiphp: Slot [29] registered [ 0.421254] acpiphp: Slot [30] registered [ 0.423144] acpiphp: Slot [31] registered [ 0.426118] PCI host bridge to bus 0000:00 [ 0.428020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.431034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.434023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.437041] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.440032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.445043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.447532] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.453000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.459460] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.468014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.471065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.473013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.477036] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.482019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.487565] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.492456] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.493082] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.494000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.494000] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.503914] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.507016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.510796] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.525019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.549018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.570016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.583557] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.591021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.603016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.624017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.636421] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.639421] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.642359] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.644482] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.647295] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.653064] iommu: Default domain type: Passthrough [ 0.654385] SCSI subsystem initialized [ 0.656196] ACPI: bus type USB registered [ 0.657267] usbcore: registered new interface driver usbfs [ 0.660090] usbcore: registered new interface driver hub [ 0.662144] usbcore: registered new device driver usb [ 0.665188] pps_core: LinuxPPS API ver. 1 registered [ 0.667018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.670095] PTP clock support registered [ 0.673207] EDAC MC: Ver: 3.0.0 [ 0.674160] PCI: Using ACPI for IRQ routing [ 0.675917] NetLabel: Initializing [ 0.676009] NetLabel: domain hash size = 128 [ 0.678013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.680117] NetLabel: unlabeled traffic allowed by default [ 0.687050] vgaarb: loaded [ 0.688183] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.691026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.701000] clocksource: Switched to clocksource kvm-clock [ 0.886765] VFS: Disk quotas dquot_6.6.0 [ 0.892900] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.898797] *** VALIDATE ramfs *** [ 0.901846] *** VALIDATE hugetlbfs *** [ 0.905378] pnp: PnP ACPI init [ 0.910942] pnp: PnP ACPI: found 6 devices [ 0.966709] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.974263] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.977908] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.980494] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.983414] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.985622] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.988535] NET: Registered protocol family 2 [ 0.993152] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.998265] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.002150] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.007920] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.012862] TCP: Hash tables configured (established 65536 bind 65536) [ 1.015974] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.021335] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.025550] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.029754] NET: Registered protocol family 1 [ 1.032125] RPC: Registered named UNIX socket transport module. [ 1.033904] RPC: Registered udp transport module. [ 1.035738] RPC: Registered tcp transport module. [ 1.037551] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.040371] NET: Registered protocol family 44 [ 1.041939] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.044104] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.046312] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.048795] PCI: CLS 0 bytes, default 64 [ 1.051275] Unpacking initramfs... [ 3.074021] hrtimer: interrupt took 10673367 ns [ 4.702831] debug: unmapping init [mem 0xffff9cc4fcc64000-0xffff9cc4fffcffff] [ 4.707863] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.713265] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.718235] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.899187] Initialise system trusted keyrings [ 6.913344] Key type blacklist registered [ 6.918643] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.952787] zbud: loaded [ 6.968829] *** VALIDATE nfs *** [ 6.970929] *** VALIDATE nfs4 *** [ 6.981038] pstore: using deflate compression [ 7.011453] Platform Keyring initialized [ 7.439978] NET: Registered protocol family 38 [ 7.442039] Key type asymmetric registered [ 7.444209] Asymmetric key parser 'x509' registered [ 7.448825] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.452855] io scheduler mq-deadline registered [ 7.457650] io scheduler kyber registered [ 7.459228] io scheduler bfq registered [ 7.467635] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.470468] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.479505] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.509179] ACPI: Power Button [PWRF] [ 7.519486] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.535461] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.555793] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.599891] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.638445] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.660525] Non-volatile memory driver v1.3 [ 7.664081] Linux agpgart interface v0.103 [ 7.834872] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 7.842954] vda: detected capacity change from 0 to 68624384 [ 7.920970] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.930535] vdb: detected capacity change from 0 to 1073741824 [ 7.942259] libphy: Fixed MDIO Bus: probed [ 7.962458] usbcore: registered new interface driver usbserial_generic [ 7.966129] usbserial: USB Serial support registered for generic [ 7.968393] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.972963] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.974739] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.976961] mousedev: PS/2 mouse device common for all mice [ 7.980406] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.982567] rtc_cmos 00:05: RTC can wake from S4 [ 8.006958] rtc_cmos 00:05: registered as rtc0 [ 8.007329] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.016723] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.022841] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.033967] intel_pstate: CPU model not supported [ 8.056142] hid: raw HID events driver (C) Jiri Kosina [ 8.058198] usbcore: registered new interface driver usbhid [ 8.062471] usbhid: USB HID core driver [ 8.067162] drop_monitor: Initializing network drop monitor service [ 8.073015] Initializing XFRM netlink socket [ 8.077418] NET: Registered protocol family 10 [ 8.083485] Segment Routing with IPv6 [ 8.084857] NET: Registered protocol family 17 [ 8.090415] mpls_gso: MPLS GSO support [ 8.100988] RAS: Correctable Errors collector initialized. [ 8.112493] AVX version of gcm_enc/dec engaged. [ 8.114107] AES CTR mode by8 optimization enabled [ 8.465957] sched_clock: Marking stable (8465938629, 0)->(9731856585, -1265917956) [ 8.475922] registered taskstats version 1 [ 8.497512] Loading compiled-in X.509 certificates [ 8.500445] zswap: loaded using pool lzo/zbud [ 8.632353] Key type big_key registered [ 8.690695] Key type encrypted registered [ 8.692138] ima: No TPM chip found, activating TPM-bypass! [ 8.694012] ima: Allocated hash algorithm: sha1 [ 8.708107] ima: No architecture policies found [ 8.709862] evm: Initialising EVM extended attributes: [ 8.718927] evm: security.selinux [ 8.720449] evm: security.ima [ 8.721913] evm: security.capability [ 8.740938] evm: HMAC attrs: 0x1 [ 8.751156] rtc_cmos 00:05: setting system clock to 2026-01-22 09:10:17 UTC (1769073017) [ 8.781149] debug: unmapping init [mem 0xffffffffb6003000-0xffffffffb61fffff] [ 8.800055] debug: unmapping init [mem 0xffffffffb4d82000-0xffffffffb5058fff] [ 8.834106] Write protecting the kernel read-only data: 28672k [ 8.863986] debug: unmapping init [mem 0xffffffffb3403000-0xffffffffb35fffff] [ 8.880865] debug: unmapping init [mem 0xffffffffb3d14000-0xffffffffb3dfffff] [ 9.068255] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.104390] systemd[1]: Detected virtualization kvm. [ 9.111635] systemd[1]: Detected architecture x86-64. [ 9.114159] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 9.193598] systemd[1]: No hostname configured. [ 9.202962] systemd[1]: Set hostname to . [ 9.212269] random: systemd: uninitialized urandom read (16 bytes read) [ 9.215431] systemd[1]: Initializing machine ID from random generator. [ 9.418256] random: ln: uninitialized urandom read (6 bytes read) [ 9.785658] random: systemd: uninitialized urandom read (16 bytes read) [ 9.810643] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 9.848847] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 9.871622] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 13.387813] device-mapper: uevent: version 1.0.3 [ 13.401510] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 16.741863] virtio_net virtio0 ens2: renamed from eth0 [ 16.937906] scsi host0: ata_piix [ 16.997753] scsi host1: ata_piix [ 17.000666] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 17.015391] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 22.686902] random: crng init done [ 22.692502] random: 7 urandom warning(s) missed due to ratelimiting [ 25.387652] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 28.870426] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 32.753472] printk: systemd: 26 output lines suppressed due to ratelimiting [ 33.317993] SELinux: Disabled at runtime. [ 33.458831] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 33.514654] systemd[1]: Detected virtualization kvm. [ 33.524526] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 34.726484] systemd[1]: initrd-switch-root.service: Succeeded. [ 34.735902] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 34.747490] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 34.753062] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 34.761673] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 34.785670] systemd[1]: Starting Journal Service... Starting Journal Service... [ 34.800346] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ 35.542860] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 36.933277] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 37.920305] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 38.023143] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 38.405219] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 38.597878] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 43.988287] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 44.847791] NFS: Registering the id_resolver key type [ 44.856437] Key type id_resolver registered [ 44.859279] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg125-client login: [ 119.425850] libcfs: loading out-of-tree module taints kernel. [ 119.780758] Key type ._llcrypt registered [ 119.787122] Key type .llcrypt registered [ 120.334916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 120.354280] alg: No test for adler32 (adler32-zlib) [ 121.673486] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 122.550608] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 124.415177] Key type lgssc registered [ 126.670366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 243.239457] Lustre: Mounted lustre-client [ 247.956567] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 263.322348] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing check_logdir /tmp/testlogs/ [ 268.183069] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing yml_node [ 268.768063] Lustre: lustre-OST0000-osc-ffff9cc543d9f000: disconnect after 23s idle [ 272.888156] Lustre: DEBUG MARKER: Client: 2.17.50.4 [ 275.577170] Lustre: DEBUG MARKER: MDS: 2.17.50.4 [ 278.657138] Lustre: DEBUG MARKER: OSS: 2.17.50.4 [ 280.230734] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Jan 22 04:14:47 EST 2026 [ 297.781026] LustreError: 7903:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9cc543d9f000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 297.795244] LustreError: 7903:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 297.830475] Lustre: Unmounted lustre-client [ 335.716237] Key type lgssc unregistered [ 336.143476] LNet: 8354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.165448] LNetError: 8354:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 336.202831] LNet: Removed LNI 192.168.201.25@tcp [ 337.377227] Key type .llcrypt unregistered [ 337.378930] Key type ._llcrypt unregistered [ 375.934981] Key type ._llcrypt registered [ 375.936989] Key type .llcrypt registered [ 376.163486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 376.177509] alg: No test for adler32 (adler32-zlib) [ 377.149732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 377.254210] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 377.318467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 382.709431] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 383.229137] LNet: 9232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.235287] LNetError: 9232:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 383.256115] LNet: Removed LNI 192.168.201.25@tcp [ 383.937380] Key type .llcrypt unregistered [ 383.941336] Key type ._llcrypt unregistered [ 391.426857] Key type ._llcrypt registered [ 391.429217] Key type .llcrypt registered [ 393.278046] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 04:16:40 (1769073400) [ 394.017274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 394.095884] alg: No test for adler32 (adler32-zlib) [ 395.275870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.443739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.501595] LNet: 9639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.565508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.690861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 395.740859] LNet: 9646:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 407.540933] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 04:16:54 (1769073414) [ 408.323481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.381972] LNet: 10224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.427131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.631123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 419.333833] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 04:17:06 (1769073426) [ 419.592904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.643913] LNet: 10811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 419.674966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.819799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 428.595189] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 04:17:15 (1769073435) [ 429.202874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.242246] LNet: 11397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 429.259681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.337378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 437.137804] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 04:17:24 (1769073444) [ 437.577617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.604468] LNet: 11982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.629593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.725983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 438.224150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 446.444987] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 04:17:33 (1769073453) [ 446.827252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.862780] LNet: 12573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 446.896877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.998278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 447.572179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 448.293916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 455.749780] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 04:17:43 (1769073463) [ 456.069794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.138633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.230618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 464.329354] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 04:17:51 (1769073471) [ 464.523699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.562663] LNet: 13751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.573424] LNet: 13751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 464.637786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.785947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 464.825842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 465.396152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 465.464849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 466.073853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 466.118193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 466.644717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 466.684626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 467.421591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 467.518970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 468.088776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 468.123270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 468.469686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 468.536951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 469.102730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 469.140529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 469.653700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 469.684734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 470.494322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 470.546460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 471.112763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 471.156899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 471.491369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 471.533775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 471.969225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 472.009730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 480.549357] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 04:18:07 (1769073487) [ 480.988939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.046497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.193919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 481.229208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 490.458781] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 04:18:17 (1769073497) [ 490.569588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.657483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.807306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 490.843642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 500.009545] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 04:18:26 (1769073506) [ 500.612466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.700685] LNet: 15612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.711191] LNet: 15612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 500.734843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.933617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 501.014623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 511.237862] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 04:18:38 (1769073518) [ 511.576604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.671243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.812823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 511.855223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 521.270089] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 04:18:48 (1769073528) [ 521.581561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.707104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.902710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 521.963740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 531.042221] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 04:18:57 (1769073537) [ 531.184097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.313524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.551065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 531.607832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 539.598544] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 04:19:06 (1769073546) [ 540.220957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.290597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.503926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 541.205461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 550.107990] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 04:19:17 (1769073557) [ 550.294330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.377171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.666410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 551.007603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 559.326836] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 04:19:26 (1769073566) [ 559.977329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.046279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.198069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 560.427968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 569.092779] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 04:19:36 (1769073576) [ 569.667680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.717646] LNet: 19788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 569.721594] LNet: 19788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 569.758217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.520813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 570.978438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 579.855591] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 04:19:46 (1769073586) [ 580.099697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.146969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.243572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 580.311132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 588.030700] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 04:19:55 (1769073595) [ 588.696735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.775537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.865325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 588.946628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 598.993841] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 04:20:05 (1769073605) [ 599.305258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.363894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.454357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 608.351512] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 04:20:15 (1769073615) [ 608.676147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.739657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.857418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 617.024046] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 04:20:24 (1769073624) [ 617.493640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.563648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.684665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 618.252705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 626.188891] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 04:20:33 (1769073633) [ 626.638287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.695127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.867761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 626.942224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 636.075142] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 04:20:43 (1769073643) [ 636.362688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.439081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.552719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 636.606747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 644.971412] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 04:20:52 (1769073652) [ 645.380598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.443801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.541135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 645.614802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 655.361943] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 04:21:02 (1769073662) [ 655.817047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.871144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.028966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 656.138609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 664.670887] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 04:21:11 (1769073671) [ 665.056663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.169937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.338856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 665.456969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 665.480565] LNetError: 25764:0:(peer.c:2225:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 665.538569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 665.602391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 674.208174] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 04:21:21 (1769073681) [ 674.929834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.983821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.088261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.S0YLA2BIqj [ 675.134676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.S0YLA2BIqj [ 675.166352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.S0YLA2BIqj [ 683.989949] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 04:21:30 (1769073690) [ 684.185928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.302656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.471184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 684.560758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 684.639510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 693.827182] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 04:21:40 (1769073700) [ 694.315829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.338319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.398866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 697.431382] LNet: Message forwarding will be enabled in 2s [ 700.511256] LNet: Message forwarding enabled [ 700.882304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 700.936408] LNet: Message forwarding will stop in 2s to 6s [ 705.583644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 706.060419] LNet: Message forwarding disabled [ 709.126980] LNet: Message forwarding will be enabled in 2s [ 712.223188] LNet: Message forwarding enabled [ 718.240861] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 04:22:05 (1769073725) [ 718.793553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.823593] LNet: 28213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.830420] LNet: 28213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 719.732318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.856704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 728.396912] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 04:22:15 (1769073735) [ 728.579412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.692634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.846756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 728.916643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 729.020088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 729.072712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 729.111381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 729.146337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 729.180619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 729.216602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 729.268709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 729.305650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 729.342275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 729.376664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 729.436824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 729.520605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 729.572228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 729.647350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 729.705880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 729.778216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 729.841587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 729.887497] LNetError: 28839:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 738.608681] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 04:22:25 (1769073745) [ 738.891846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 738.956470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.156434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 748.638477] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 04:22:35 (1769073755) [ 751.637124] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg125-server [ 754.564738] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 04:22:41 (1769073761) [ 755.465500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 755.580245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.713760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 755.829625] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 757.888504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[26-30]@tcp [ 757.935321] LNet: 30557:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 757.996517] LNetError: 30558:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.29@tcp has gone from up to down [ 758.047558] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.26@tcp is being used as a gateway but routing feature is not turned on [ 759.016438] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.30@tcp is being used as a gateway but routing feature is not turned on [ 759.025071] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 760.045571] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.27@tcp is being used as a gateway but routing feature is not turned on [ 760.088024] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 762.093164] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.26@tcp is being used as a gateway but routing feature is not turned on [ 762.115661] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 766.187707] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.27@tcp is being used as a gateway but routing feature is not turned on [ 766.195359] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 767.604122] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 04:22:54 (1769073774) [ 769.749962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg125-server [ 772.529633] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 04:22:58 (1769073778) [ 772.670922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 772.736750] LNetError: 31486:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 772.775548] LNet: Removed LNI 192.168.201.25@tcp [ 772.877993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.024799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 773.065580] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 774.310885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[26-31/2]@tcp [ 774.332244] LNet: 31612:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 774.343126] LNet: 31612:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 774.380841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[26-31/2]@tcp [ 774.390695] LNetError: 31498:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.26@tcp is being used as a gateway but routing feature is not turned on [ 774.432244] LNetError: 31498:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 27 previous similar messages [ 774.450618] LNetError: 31498:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.26@tcp has gone from up to down [ 774.470710] LNetError: 31498:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 784.305118] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 04:23:11 (1769073791) [ 784.475552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.527055] LNetError: 32200:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.562284] LNet: Removed LNI 192.168.201.25@tcp [ 784.617547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.737715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 784.802870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.843588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.927556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 785.013337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.093851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.177037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 785.283468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.360890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.489293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 785.608964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.660971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.715139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 785.833576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.920587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.006123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 786.181356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.255621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.310450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 786.388967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.425273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.497986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 786.584805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.625280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.667084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 786.736852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.762662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.806805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 786.852743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.881786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.923701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 786.979629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.018703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.078590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 795.119406] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 04:23:21 (1769073801) [ 797.342390] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg125-server [ 799.860308] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 04:23:26 (1769073806) [ 801.890543] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg125-server [ 804.246122] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 04:23:31 (1769073811) [ 805.492972] Key type .llcrypt unregistered [ 805.495438] Key type ._llcrypt unregistered [ 813.597602] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 813.748228] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 813.761393] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 813.768601] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 816.673509] Key type ._llcrypt registered [ 816.682517] Key type .llcrypt registered [ 816.954444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 816.968308] alg: No test for adler32 (adler32-zlib) [ 817.959377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.041463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 818.071181] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 819.037294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 819.072585] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 822.070868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 822.103392] LNet: Removed LNI 10.1.2.3@tcp [ 822.296980] LNet: 34331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.322905] LNetError: 34331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 822.353766] LNet: Removed LNI 192.168.201.25@tcp [ 823.168691] Key type .llcrypt unregistered [ 823.170738] Key type ._llcrypt unregistered [ 840.594152] Key type ._llcrypt registered [ 840.597159] Key type .llcrypt registered [ 843.115950] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 04:24:08 (1769073848) [ 844.157404] Key type .llcrypt unregistered [ 844.160504] Key type ._llcrypt unregistered [ 854.313459] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 854.552339] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 854.566096] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 854.576285] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 857.632247] Key type ._llcrypt registered [ 857.633976] Key type .llcrypt registered [ 858.061962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 858.137675] alg: No test for adler32 (adler32-zlib) [ 859.220923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.346155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 859.400844] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 860.709436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 860.741643] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 864.512717] LNet: 35818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.518488] LNetError: 35818:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.528370] LNet: Removed LNI 192.168.201.25@tcp [ 864.541764] LNet: Removed LNI 10.1.2.3@tcp [ 865.398682] Key type .llcrypt unregistered [ 865.401103] Key type ._llcrypt unregistered [ 881.493175] Key type ._llcrypt registered [ 881.496431] Key type .llcrypt registered [ 883.038261] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 04:24:50 (1769073890) [ 883.938299] Key type .llcrypt unregistered [ 883.939754] Key type ._llcrypt unregistered [ 891.655289] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 891.892896] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 891.906087] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 891.919744] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 894.620376] Key type ._llcrypt registered [ 894.625671] Key type .llcrypt registered [ 894.842094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 894.872387] alg: No test for adler32 (adler32-zlib) [ 895.905951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.011301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 896.048304] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 898.654891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 898.676572] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 901.580207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 901.633876] LNet: Removed LNI 10.1.2.3@tcp [ 901.663926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 901.705396] LNet: Removed LNI 10.1.2.31@tcp [ 901.739631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 901.769743] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 902.896360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 902.921975] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 904.251566] LNet: 37729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.259482] LNetError: 37729:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 904.275583] LNet: Removed LNI 10.1.2.3@tcp [ 904.288679] LNet: Removed LNI 10.1.2.31@tcp [ 905.145560] Key type .llcrypt unregistered [ 905.147976] Key type ._llcrypt unregistered [ 921.979207] Key type ._llcrypt registered [ 921.980784] Key type .llcrypt registered [ 923.910150] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 04:25:30 (1769073930) [ 924.432357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 924.457315] alg: No test for adler32 (adler32-zlib) [ 925.475184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.608050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 925.660591] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 926.865452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.913700] LNet: 38679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.925627] LNetError: 38679:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.945264] LNet: Removed LNI 192.168.201.25@tcp [ 927.005533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.112390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.25@tcp [ 927.141588] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 928.167314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.25@tcp [ 928.200730] LNet: Removed LNI 192.168.201.25@tcp [ 928.280375] LNet: 38814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.033662] Key type .llcrypt unregistered [ 929.037720] Key type ._llcrypt unregistered [ 943.531623] Key type ._llcrypt registered [ 943.534522] Key type .llcrypt registered [ 945.257444] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 04:25:52 (1769073952) [ 947.728160] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg125-server [ 949.748067] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 04:25:56 (1769073956) [ 950.602608] Key type .llcrypt unregistered [ 950.610279] Key type ._llcrypt unregistered [ 958.439227] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 958.697741] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 958.719891] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 958.732787] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 961.744605] Key type ._llcrypt registered [ 961.751778] Key type .llcrypt registered [ 962.005508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 962.038887] alg: No test for adler32 (adler32-zlib) [ 963.056956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.149776] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 963.976912] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 966.321711] LNet: 40660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.331672] LNetError: 40660:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.339377] LNet: Removed LNI 192.168.201.25@tcp1 [ 966.355970] LNet: Removed LNI 10.1.2.3@tcp2 [ 967.192376] Key type .llcrypt unregistered [ 967.194570] Key type ._llcrypt unregistered [ 982.801545] Key type ._llcrypt registered [ 982.805602] Key type .llcrypt registered [ 984.608418] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 04:26:31 (1769073991) [ 985.068920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 985.087869] alg: No test for adler32 (adler32-zlib) [ 986.094684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.174942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 986.204099] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 987.080451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.114486] LNet: 41625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.123583] LNetError: 41625:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.143050] LNet: Removed LNI 192.168.201.25@tcp [ 987.164474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.233793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 987.267842] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 996.764371] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 04:26:43 (1769074003) [ 997.025086] LNet: 42216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.067828] LNetError: 42216:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.093462] LNet: Removed LNI 192.168.201.25@tcp [ 997.838255] Key type .llcrypt unregistered [ 997.844278] Key type ._llcrypt unregistered [ 1006.420457] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1006.596854] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1006.604419] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1006.612488] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1009.485722] Key type ._llcrypt registered [ 1009.492956] Key type .llcrypt registered [ 1009.756173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.786959] alg: No test for adler32 (adler32-zlib) [ 1010.793562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.883483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1010.918722] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1011.910961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1011.940228] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1014.145521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1014.171236] LNet: 42955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.176472] LNetError: 42955:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.183860] LNet: Removed LNI 192.168.201.25@tcp [ 1014.216683] LNet: Removed LNI 10.1.2.3@tcp [ 1014.274855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.375159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1014.431688] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1014.439936] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1015.044106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1015.070372] LNet: 42982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.080322] LNetError: 42982:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.086635] LNetError: 42982:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1015.093968] LNet: Removed LNI 192.168.201.25@tcp [ 1015.106827] LNet: Removed LNI 10.1.2.3@tcp [ 1015.136334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.238749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1015.280251] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1015.285111] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1015.851718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1015.887596] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1016.765263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1016.785979] LNet: 43145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.795862] LNetError: 43145:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.803200] LNetError: 43145:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1016.811345] LNet: Removed LNI 192.168.201.25@tcp [ 1016.813146] LNet: Removed LNI 10.1.2.3@tcp [ 1016.831051] LNet: Removed LNI 192.168.201.25@tcp2 [ 1016.874792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1016.976785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1017.011327] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1017.020181] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1017.038565] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1027.950535] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 04:27:14 (1769074034) [ 1028.101811] LNet: 43733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.117834] LNetError: 43733:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.131416] LNetError: 43733:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1028.147748] LNet: Removed LNI 192.168.201.25@tcp [ 1028.164783] LNet: Removed LNI 10.1.2.3@tcp [ 1028.185472] LNet: Removed LNI 192.168.201.25@tcp2 [ 1029.193768] Key type .llcrypt unregistered [ 1029.197496] Key type ._llcrypt unregistered [ 1038.401425] Key type ._llcrypt registered [ 1038.410187] Key type .llcrypt registered [ 1038.715350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1038.750586] alg: No test for adler32 (adler32-zlib) [ 1039.792473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1039.883470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1039.930646] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1041.087955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1041.108810] LNet: 44133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.114913] LNetError: 44133:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.125820] LNet: Removed LNI 192.168.201.25@tcp [ 1041.156651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.222775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1041.253169] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1041.807594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1041.844563] LNet: 44163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.852714] LNetError: 44163:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.866336] LNet: Removed LNI 192.168.201.25@tcp [ 1041.881511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.964759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1041.997579] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1042.692786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.729964] LNet: Removed LNI 192.168.201.25@tcp [ 1042.759770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.825544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1042.849691] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1043.557089] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1043.798903] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1043.807616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1043.818722] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1046.261806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1046.289358] LNet: 44268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.300700] LNet: 44268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1046.312517] LNetError: 44268:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.315863] LNetError: 44268:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1046.333068] LNet: Removed LNI 192.168.201.25@tcp [ 1046.359490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1046.476580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1046.518682] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1055.456214] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 04:27:42 (1769074062) [ 1055.582594] LNet: 44856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.591958] LNetError: 44856:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1055.617354] LNet: Removed LNI 192.168.201.25@tcp [ 1056.249222] Key type .llcrypt unregistered [ 1056.250856] Key type ._llcrypt unregistered [ 1064.979850] Key type ._llcrypt registered [ 1064.982104] Key type .llcrypt registered [ 1065.197529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1065.213082] alg: No test for adler32 (adler32-zlib) [ 1066.321906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1066.447324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1066.471466] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1067.681947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1067.703734] LNet: 45259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.708978] LNetError: 45259:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.722597] LNet: Removed LNI 192.168.201.25@tcp [ 1067.754041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1067.833155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1067.896693] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1076.455687] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 04:28:03 (1769074083) [ 1077.182451] LNet: 45851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.193962] LNetError: 45851:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.220851] LNet: Removed LNI 192.168.201.25@tcp [ 1078.022507] Key type .llcrypt unregistered [ 1078.025761] Key type ._llcrypt unregistered [ 1084.894499] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1085.017448] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1085.021515] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1085.023631] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1087.858426] Key type ._llcrypt registered [ 1087.871807] Key type .llcrypt registered [ 1088.151834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1088.165672] alg: No test for adler32 (adler32-zlib) [ 1089.230067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1089.381316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1089.435940] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1090.784292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1090.827886] LNet: 46292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1090.836626] LNetError: 46292:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1090.850872] LNet: Removed LNI 192.168.201.25@tcp [ 1090.909963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1091.032975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1091.092888] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1100.448769] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 04:28:27 (1769074107) [ 1100.858765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1100.892499] LNet: 46892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.932414] LNetError: 46892:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1100.963611] LNet: Removed LNI 192.168.201.25@tcp [ 1101.034379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1101.275457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1110.262843] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 04:28:36 (1769074116) [ 1110.384640] LNet: 47466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1111.482869] Key type .llcrypt unregistered [ 1111.488767] Key type ._llcrypt unregistered [ 1119.596675] Key type ._llcrypt registered [ 1119.598453] Key type .llcrypt registered [ 1119.961249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1120.002968] alg: No test for adler32 (adler32-zlib) [ 1121.110300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1121.195372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1121.231507] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1122.275283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1122.311559] LNet: 47869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1122.322832] LNetError: 47869:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1122.344927] LNet: Removed LNI 192.168.201.25@tcp [ 1122.386480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1122.467202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1122.525105] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1131.462961] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 04:28:58 (1769074138) [ 1132.067772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1132.136067] LNet: 48462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1132.158089] LNetError: 48462:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1132.185909] LNet: Removed LNI 192.168.201.25@tcp [ 1132.258347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1132.442902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1140.032772] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 04:29:06 (1769074146) [ 1142.389855] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1145.839554] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 04:29:11 (1769074151) [ 1147.908793] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1149.908444] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 04:29:16 (1769074156) [ 1151.544474] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1154.311727] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 04:29:20 (1769074160) [ 1156.099533] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1158.428458] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 04:29:25 (1769074165) [ 1160.436342] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1162.813482] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 04:29:29 (1769074169) [ 1162.995030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1163.037329] LNet: 50514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.098687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1163.184204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1163.208663] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1164.368603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.25 [ 1164.387659] LNet: 50651:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1164.398054] LNet: Removed LNI 192.168.201.25@tcp [ 1172.101590] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 04:29:39 (1769074179) [ 1172.532503] LNet: 51222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.539916] LNetError: 51222:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.561546] LNet: Removed LNI 192.168.201.25@tcp [ 1173.352389] Key type .llcrypt unregistered [ 1173.364787] Key type ._llcrypt unregistered [ 1180.421736] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1180.739216] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1180.748851] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1180.769956] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1183.688896] Key type ._llcrypt registered [ 1183.691477] Key type .llcrypt registered [ 1184.047561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1184.081952] alg: No test for adler32 (adler32-zlib) [ 1185.574469] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1194.035142] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 04:30:00 (1769074200) [ 1194.185100] LNet: 52115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1194.203497] LNetError: 52115:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1194.255342] LNet: Removed LNI 10.1.2.3@tcp [ 1195.182810] Key type .llcrypt unregistered [ 1195.188899] Key type ._llcrypt unregistered [ 1202.206432] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1202.495548] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1202.498771] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1202.501712] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1205.377110] Key type ._llcrypt registered [ 1205.378137] Key type .llcrypt registered [ 1205.618427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1205.643230] alg: No test for adler32 (adler32-zlib) [ 1206.517916] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1215.527661] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 04:30:23 (1769074223) [ 1216.118783] LNet: 53013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.125607] LNetError: 53013:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1216.165684] LNet: Removed LNI 10.1.2.3@tcp [ 1216.922768] Key type .llcrypt unregistered [ 1216.924286] Key type ._llcrypt unregistered [ 1224.577377] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1224.883470] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1224.885791] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1224.891531] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1227.927442] Key type ._llcrypt registered [ 1227.949990] Key type .llcrypt registered [ 1228.263336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1228.279430] alg: No test for adler32 (adler32-zlib) [ 1229.337959] LNet: 53310:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1229.376654] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1237.250805] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 04:30:44 (1769074244) [ 1237.730467] LNet: 53902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.740966] LNetError: 53902:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1237.787204] LNet: Removed LNI 10.1.2.3@tcp [ 1238.958128] Key type .llcrypt unregistered [ 1238.961160] Key type ._llcrypt unregistered [ 1248.527817] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1248.827564] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1248.830473] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1248.836226] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1251.691977] Key type ._llcrypt registered [ 1251.699650] Key type .llcrypt registered [ 1251.963586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1251.998962] alg: No test for adler32 (adler32-zlib) [ 1252.989588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1253.086105] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1263.205755] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 04:31:10 (1769074270) [ 1263.976944] LNet: 55128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.989152] LNetError: 55128:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1264.000994] LNet: Removed LNI 10.1.2.3@tcp [ 1264.737348] Key type .llcrypt unregistered [ 1264.746776] Key type ._llcrypt unregistered [ 1272.591492] Key type ._llcrypt registered [ 1272.600370] Key type .llcrypt registered [ 1272.946416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1272.987379] alg: No test for adler32 (adler32-zlib) [ 1274.019748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1274.082858] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1282.733627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1283.782913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_interrupt -r 1 [ 1283.885942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_interrupt -r 1 [ 1283.947750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1283.975701] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1284.319926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1284.328470] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 1284.379377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1284.445470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1284.479490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1284.531443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1284.590334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1284.660427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1284.702393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1284.721661] LNet: 55562:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1284.746428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1284.789082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1284.852795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1284.923885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1285.013729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1285.094570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1285.144461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1285.751207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_dropped -r 1 [ 1285.785459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_dropped -r 1 [ 1285.847659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1285.884444] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1285.895909] LNet: Skipped 1 previous similar message [ 1286.119263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1286.173106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1286.236820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1286.268883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1286.310803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1286.347816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1286.409366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1286.467225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1286.498948] LNet: 55671:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1286.533567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1286.585573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1286.668106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1286.726698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1286.781641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1286.833131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1286.891423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1287.524929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_aborted -r 1 [ 1287.567766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_aborted -r 1 [ 1287.636227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1287.658954] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1287.931258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1287.964568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1287.997567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1288.036340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1288.076023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1288.136988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1288.185541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1288.241350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1288.263913] LNet: 55780:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1288.290209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1288.326501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1288.377484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1288.416266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1288.454362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.521767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1288.562291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1289.118051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_no_route -r 1 [ 1289.204716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_no_route -r 1 [ 1289.265622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1289.545701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1289.574368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1289.601511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1289.629802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1289.661865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1289.694860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1289.727620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1289.753412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1289.776439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1289.805362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1289.834403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1289.860508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1289.884410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1289.918340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1289.944376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1290.527257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_timeout -r 1 [ 1290.573262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_timeout -r 1 [ 1290.621196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1290.641746] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1290.645516] LNet: Skipped 2 previous similar messages [ 1290.865454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1290.916657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1290.944597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1290.998861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1291.046485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1291.077449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1291.114521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1291.151729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1291.172940] LNet: 55999:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1291.193875] LNet: 55999:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1291.223215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1291.279082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1291.334809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1291.395686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1291.441131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1291.495405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1291.561113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1292.177747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_error -r 1 [ 1292.214551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_error -r 1 [ 1292.246971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1292.564707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1292.638377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1292.701353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1292.751876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1292.799980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1292.865821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1292.916442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1292.964774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1293.042416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1293.136540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1293.191698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1293.231706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1293.289821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1293.363906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1293.425546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1293.894325] LNet: 56148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1293.917604] LNetError: 56148:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.947197] LNet: Removed LNI 192.168.201.25@tcp [ 1294.563301] Key type .llcrypt unregistered [ 1294.576311] Key type ._llcrypt unregistered [ 1315.271339] Key type ._llcrypt registered [ 1315.284182] Key type .llcrypt registered [ 1317.164841] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 04:32:03 (1769074323) [ 1318.077386] Key type .llcrypt unregistered [ 1318.079271] Key type ._llcrypt unregistered [ 1326.151513] Key type ._llcrypt registered [ 1326.157230] Key type .llcrypt registered [ 1326.447975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1326.467407] alg: No test for adler32 (adler32-zlib) [ 1327.552416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1327.595662] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1336.880356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1349.131687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1349.182983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1349.202661] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1350.890182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_interrupt -r 1 [ 1350.938676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_interrupt -r 1 [ 1350.990315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_interrupt -r 1 [ 1351.064480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_interrupt -r 1 [ 1351.113900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_interrupt -r 1 [ 1351.164730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_interrupt -r 1 [ 1351.235949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_interrupt -r 1 [ 1351.288634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_interrupt -r 1 [ 1351.360144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1351.406270] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1351.417801] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 1351.798899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1351.828229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1351.862886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1351.917265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1351.954346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1351.979902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1352.002470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1352.026928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1352.043856] LNet: 57612:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1352.063318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1352.090282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1352.114354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1352.163695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1352.203638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1352.247708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1352.308222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1353.080661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_dropped -r 1 [ 1353.120774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_dropped -r 1 [ 1353.179776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_dropped -r 1 [ 1353.232640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_dropped -r 1 [ 1353.307627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_dropped -r 1 [ 1353.353760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_dropped -r 1 [ 1353.403653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_dropped -r 1 [ 1353.475628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_dropped -r 1 [ 1353.549352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1353.579315] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1353.588578] LNet: Skipped 2 previous similar messages [ 1353.881617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1353.919591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1353.954650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1353.980485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1354.007695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1354.055454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1354.102736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1354.147879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1354.172612] LNet: 57734:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1354.198775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1354.251657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1354.298461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1354.347984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1354.380456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1354.489139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1354.539937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1355.067789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_aborted -r 1 [ 1355.116648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_aborted -r 1 [ 1355.168297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_aborted -r 1 [ 1355.248627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_aborted -r 1 [ 1355.310714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_aborted -r 1 [ 1355.387849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_aborted -r 1 [ 1355.456444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_aborted -r 1 [ 1355.486947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_aborted -r 1 [ 1355.554855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1355.572382] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1355.581646] LNet: Skipped 2 previous similar messages [ 1355.881610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1355.922707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1355.966224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1356.003489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1356.059967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1356.103677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1356.148849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1356.192641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1356.207764] LNet: 57855:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1356.241696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1356.286541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1356.333665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1356.378679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1356.429556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1356.480290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1356.531705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1357.478144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_no_route -r 1 [ 1357.532855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_no_route -r 1 [ 1357.572862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_no_route -r 1 [ 1357.621622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_no_route -r 1 [ 1357.665791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_no_route -r 1 [ 1357.697972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_no_route -r 1 [ 1357.749845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_no_route -r 1 [ 1357.783321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_no_route -r 1 [ 1357.819258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1357.836841] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1357.841818] LNet: Skipped 2 previous similar messages [ 1358.160721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1358.193676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1358.235506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1358.277534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1358.307418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1358.353292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1358.396931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1358.448982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1358.481887] LNet: 57977:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1358.513598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1358.554941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1358.599264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1358.662991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1358.720910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1358.764742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1358.794747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1359.410599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_timeout -r 1 [ 1359.459824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_timeout -r 1 [ 1359.508937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_timeout -r 1 [ 1359.554490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_timeout -r 1 [ 1359.580174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_timeout -r 1 [ 1359.639815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_timeout -r 1 [ 1359.708664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_timeout -r 1 [ 1359.774533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_timeout -r 1 [ 1359.845826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1360.224376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1360.291452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1360.333179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1360.373919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1360.420757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1360.447830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1360.479966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1360.505566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1360.533630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1360.561507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1360.596547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1360.621859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1360.654344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1360.696396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1360.740705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1361.415228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_error -r 1 [ 1361.476570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_error -r 1 [ 1361.572642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_error -r 1 [ 1361.640420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_error -r 1 [ 1361.701326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_error -r 1 [ 1361.812346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_error -r 1 [ 1361.892813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_error -r 1 [ 1361.944682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_error -r 1 [ 1361.976705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1362.019800] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1362.030216] LNet: Skipped 5 previous similar messages [ 1362.341028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1362.396441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1362.449442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1362.501675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1362.544642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1362.592600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1362.671418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1362.723922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1362.764778] LNet: 58219:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1362.772459] LNet: 58219:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1362.808357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1362.852726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1362.905262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1362.954209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1362.999164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1363.050152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1363.110689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1369.989342] LNet: 58295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1369.998173] LNetError: 58295:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1370.005262] LNet: Removed LNI 192.168.201.25@tcp [ 1370.019184] LNet: Removed LNI 192.168.201.25@tcp1 [ 1370.524560] Key type .llcrypt unregistered [ 1370.526770] Key type ._llcrypt unregistered [ 1390.272634] Key type ._llcrypt registered [ 1390.282406] Key type .llcrypt registered [ 1392.015711] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 04:33:18 (1769074398) [ 1392.972910] Key type .llcrypt unregistered [ 1392.975227] Key type ._llcrypt unregistered [ 1401.225758] Key type ._llcrypt registered [ 1401.229838] Key type .llcrypt registered [ 1401.557856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1401.609247] alg: No test for adler32 (adler32-zlib) [ 1402.632812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1402.700951] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1412.985892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1414.031600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e remote_dropped -r 1 [ 1414.085070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e remote_dropped -r 1 [ 1414.130696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1414.152901] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1414.539472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1414.599465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1414.676728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1414.722803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1414.776461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1414.809718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1414.860395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1414.908657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1414.935644] LNet: 59541:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1414.974732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1415.021588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1415.080489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1415.128573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1415.181102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1415.214161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1415.273538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1416.049203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e remote_error -r 1 [ 1416.084618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e remote_error -r 1 [ 1416.151924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1416.172320] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1416.478452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1416.524592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1416.570498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1416.626566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1416.688224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1416.771897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1416.857106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1416.915227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1416.950848] LNet: 59650:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1416.984374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1417.042739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1417.097378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1417.151499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1417.214614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1417.270532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1417.328965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1418.168728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e remote_timeout -r 1 [ 1418.214510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e remote_timeout -r 1 [ 1418.293888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1418.339051] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1418.643634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1418.697975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1418.744931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1418.779618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1418.806708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1418.868939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1418.929207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1418.984346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1419.016372] LNet: 59759:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1419.044673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1419.099987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1419.153806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1419.196578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1419.269846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1419.353600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1419.433579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1419.811324] LNet: 59800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1419.825413] LNetError: 59800:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1419.855527] LNet: Removed LNI 192.168.201.25@tcp [ 1420.393613] Key type .llcrypt unregistered [ 1420.397191] Key type ._llcrypt unregistered [ 1440.430196] Key type ._llcrypt registered [ 1440.435646] Key type .llcrypt registered [ 1442.120956] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 04:34:09 (1769074449) [ 1443.383559] Key type .llcrypt unregistered [ 1443.388717] Key type ._llcrypt unregistered [ 1451.184933] Key type ._llcrypt registered [ 1451.189632] Key type .llcrypt registered [ 1451.550997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1451.592888] alg: No test for adler32 (adler32-zlib) [ 1452.628252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1452.659964] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1461.377129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1471.481352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.535030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1471.568956] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1473.551763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e remote_dropped -r 1 [ 1473.615404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e remote_dropped -r 1 [ 1473.657582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e remote_dropped -r 1 [ 1473.704969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e remote_dropped -r 1 [ 1473.737968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e remote_dropped -r 1 [ 1473.769775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e remote_dropped -r 1 [ 1473.803366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e remote_dropped -r 1 [ 1473.833377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e remote_dropped -r 1 [ 1473.867587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1473.899938] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1473.912682] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.125@tcp [ 1474.125720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1474.147105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1474.176497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1474.201374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1474.228682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1474.254952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1474.279747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1474.305354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1474.322845] LNet: 61266:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1474.341454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1474.370658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1474.398954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1474.447914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1474.478864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1474.544850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1474.620781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1474.664359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1475.382397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e remote_error -r 1 [ 1475.424689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e remote_error -r 1 [ 1475.502426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e remote_error -r 1 [ 1475.574551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e remote_error -r 1 [ 1475.656769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e remote_error -r 1 [ 1475.708282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e remote_error -r 1 [ 1475.748507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e remote_error -r 1 [ 1475.785899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e remote_error -r 1 [ 1475.842880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1475.862664] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1475.867425] LNet: Skipped 2 previous similar messages [ 1476.152521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1476.179515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1476.217893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1476.255638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1476.329788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1476.406927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1476.451318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1476.499205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1476.511275] LNet: 61390:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1476.526490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1476.580445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1476.621543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1476.675359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1476.726882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1476.781853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1476.832968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1476.895989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1477.477661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e remote_timeout -r 1 [ 1477.508470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e remote_timeout -r 1 [ 1477.528399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e remote_timeout -r 1 [ 1477.547218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e remote_timeout -r 1 [ 1477.574786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e remote_timeout -r 1 [ 1477.600609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e remote_timeout -r 1 [ 1477.633217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e remote_timeout -r 1 [ 1477.658450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e remote_timeout -r 1 [ 1477.693976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1477.707549] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1477.920806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1477.953906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1477.977195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1478.002920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1478.037615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1478.076405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1478.115137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1478.151627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1478.184909] LNet: 61513:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1478.203927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1478.250705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1478.289144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1478.339716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1478.380500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1478.418151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1478.479524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1478.535844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1484.739078] LNet: 61590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.749177] LNetError: 61590:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1484.763659] LNet: Removed LNI 192.168.201.25@tcp [ 1484.800716] LNet: Removed LNI 192.168.201.25@tcp1 [ 1485.664833] Key type .llcrypt unregistered [ 1485.671427] Key type ._llcrypt unregistered [ 1506.654202] Key type ._llcrypt registered [ 1506.655831] Key type .llcrypt registered [ 1508.438379] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 04:35:15 (1769074515) [ 1509.147300] Key type .llcrypt unregistered [ 1509.152734] Key type ._llcrypt unregistered [ 1517.600721] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1517.777976] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1517.790779] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1517.799410] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1520.686967] Key type ._llcrypt registered [ 1520.689514] Key type .llcrypt registered [ 1520.902853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1520.922592] alg: No test for adler32 (adler32-zlib) [ 1521.964244] LNet: 62665:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1522.023165] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1522.122872] LNet: 62679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1522.144632] LNetError: 62679:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1522.160423] LNet: Removed LNI 192.168.201.25@tcp [ 1522.808523] Key type .llcrypt unregistered [ 1522.812668] Key type ._llcrypt unregistered [ 1529.216382] Key type ._llcrypt registered [ 1529.219463] Key type .llcrypt registered [ 1529.520869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1529.553132] alg: No test for adler32 (adler32-zlib) [ 1530.581964] LNet: 62934:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1530.621289] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1530.629366] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1530.695857] LNet: 62948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.704541] LNetError: 62948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.722833] LNet: Removed LNI 192.168.201.25@tcp [ 1530.745529] LNet: Removed LNI 10.1.2.3@tcp1 [ 1531.637542] Key type .llcrypt unregistered [ 1531.639588] Key type ._llcrypt unregistered [ 1540.119763] Key type ._llcrypt registered [ 1540.122798] Key type .llcrypt registered [ 1540.503265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1540.523778] alg: No test for adler32 (adler32-zlib) [ 1541.567697] LNet: 63204:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1541.583934] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1541.589342] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1541.632385] LNet: 63218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.639374] LNetError: 63218:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1541.642753] LNet: Removed LNI 192.168.201.25@tcp [ 1541.659798] LNet: Removed LNI 10.1.2.3@tcp [ 1542.503374] Key type .llcrypt unregistered [ 1542.505067] Key type ._llcrypt unregistered [ 1551.915291] Key type ._llcrypt registered [ 1551.918241] Key type .llcrypt registered [ 1552.131764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1552.147623] alg: No test for adler32 (adler32-zlib) [ 1553.158620] LNet: 63476:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1553.179312] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1553.209573] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1553.322494] LNet: 63490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.328694] LNetError: 63490:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1553.336713] LNet: Removed LNI 192.168.201.25@tcp [ 1553.349939] LNet: Removed LNI 10.1.2.3@tcp [ 1554.350542] Key type .llcrypt unregistered [ 1554.352142] Key type ._llcrypt unregistered [ 1562.911607] Key type ._llcrypt registered [ 1562.913995] Key type .llcrypt registered [ 1563.106350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1563.123346] alg: No test for adler32 (adler32-zlib) [ 1564.156974] LNet: 63751:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1564.189089] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1564.195919] LNetError: 63751:0:(config.c:1560:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1564.203986] LNetError: Error -100 starting up LNI tcp [ 1564.216063] LNet: Removed LNI 192.168.201.25@tcp [ 1565.192591] Key type .llcrypt unregistered [ 1565.204380] Key type ._llcrypt unregistered [ 1579.183302] Key type ._llcrypt registered [ 1579.189959] Key type .llcrypt registered [ 1580.768429] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 04:36:27 (1769074587) [ 1581.510352] Key type .llcrypt unregistered [ 1581.516393] Key type ._llcrypt unregistered [ 1589.640382] Key type ._llcrypt registered [ 1589.645517] Key type .llcrypt registered [ 1589.918450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1589.945760] alg: No test for adler32 (adler32-zlib) [ 1590.898701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1590.926139] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1600.564925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1601.437801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e network_timeout -r 1 [ 1601.498740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e network_timeout -r 1 [ 1601.537309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1601.578729] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1601.782961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1601.808363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1601.837751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1601.853568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1601.873816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1601.918741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1601.936124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1601.958261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1601.971305] LNet: 65003:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1601.982554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1602.003517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1602.019850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1602.038499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1602.061797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1602.080035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1602.341202] LNet: 65041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1602.350158] LNetError: 65041:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.408515] LNet: Removed LNI 192.168.201.25@tcp [ 1604.007592] Key type .llcrypt unregistered [ 1604.010908] Key type ._llcrypt unregistered [ 1624.766865] Key type ._llcrypt registered [ 1624.769671] Key type .llcrypt registered [ 1625.211708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1625.252649] alg: No test for adler32 (adler32-zlib) [ 1626.405963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1626.465959] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1636.615723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1647.081965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.156588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1647.206454] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1648.674679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e network_timeout -r 1 [ 1648.721733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e network_timeout -r 1 [ 1648.755709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e network_timeout -r 1 [ 1648.817460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e network_timeout -r 1 [ 1648.860749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e network_timeout -r 1 [ 1648.907895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e network_timeout -r 1 [ 1648.953073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e network_timeout -r 1 [ 1649.029189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e network_timeout -r 1 [ 1649.113096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1649.156753] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1649.422993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1649.472794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1649.524352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1649.556770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1649.580715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1649.601728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1649.639892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1649.677619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1649.716297] LNet: 65918:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1649.751032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1649.788481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1649.822358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1649.867641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1649.931904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1649.964358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1654.930046] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1655.139323] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 1655.510236] LNet: 65991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.519956] LNetError: 65991:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.533542] LNet: Removed LNI 192.168.201.25@tcp [ 1655.552734] LNet: Removed LNI 192.168.201.25@tcp1 [ 1655.948914] Key type .llcrypt unregistered [ 1655.953682] Key type ._llcrypt unregistered [ 1674.100979] Key type ._llcrypt registered [ 1674.102938] Key type .llcrypt registered [ 1675.490628] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 04:38:02 (1769074682) [ 1675.994784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1676.035178] alg: No test for adler32 (adler32-zlib) [ 1677.031873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1677.127664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1677.163314] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1677.890143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1677.919931] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1678.828554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1678.892406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1678.940474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1678.995438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1679.067718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1679.123429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1679.158966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.25@tcp [ 1679.181746] LNetError: 67088:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.201.25@tcp to recovery [ 1680.161621] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 1680.164158] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1682.214276] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1682.727352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1686.303462] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1687.379238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1687.439632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1687.541937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.565627] LNet: 67206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.576608] LNetError: 67206:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.607163] LNet: Removed LNI 192.168.201.25@tcp [ 1688.611376] LNetError: 67206:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.631788] LNet: Removed LNI 192.168.201.25@tcp1 [ 1688.673323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.775498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1688.806515] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1689.602443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1689.625101] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1690.526359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1690.577077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1690.688397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1690.778135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1690.836830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1690.877607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1690.919808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1690.943962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.25@tcp [ 1690.962427] LNetError: 67478:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.201.25@tcp to recovery [ 1691.809167] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1694.470858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1697.954239] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1697.964895] LNet: Skipped 1 previous similar message [ 1701.433981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1701.478723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1701.523317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1710.133413] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 04:38:37 (1769074717) [ 1710.266193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.299570] LNet: 68182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.307315] LNetError: 68182:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.315124] LNet: Removed LNI 192.168.201.25@tcp [ 1710.334437] LNet: Removed LNI 192.168.201.25@tcp1 [ 1710.384857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.471456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1710.493995] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1711.362499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1711.385139] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1712.645869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1712.715230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1712.744914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1712.784486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1712.819932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1712.850837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1712.882686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.25@tcp --health 0 [ 1712.992667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1713.503467] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.25@tcp [ 1713.505340] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1713.520504] LNet: Skipped 1 previous similar message [ 1728.847178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1729.084140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1729.123962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1729.166564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.25@tcp --health 500 [ 1729.291866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1736.126917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1736.248379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.287437] LNet: 68680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.295174] LNetError: 68680:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.303056] LNetError: 68680:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1736.314363] LNet: Removed LNI 192.168.201.25@tcp [ 1736.333705] LNet: Removed LNI 192.168.201.25@tcp1 [ 1736.368133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.506609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1736.545361] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1737.690532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1737.725648] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1738.914627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1738.947455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1739.034542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1739.094671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1739.117818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1739.144685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1739.178312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1739.202774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.25@tcp --health 0 [ 1740.512464] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1740.517944] LNet: Skipped 3 previous similar messages [ 1741.701892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1751.085276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1751.143969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1751.206982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1759.711782] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 04:39:27 (1769074767) [ 1760.409864] LNet: 69726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.428906] LNetError: 69726:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1760.440180] LNetError: 69726:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1760.457737] LNet: Removed LNI 192.168.201.25@tcp [ 1760.474767] LNet: Removed LNI 192.168.201.25@tcp1 [ 1761.308623] Key type .llcrypt unregistered [ 1761.310246] Key type ._llcrypt unregistered [ 1768.542522] Key type ._llcrypt registered [ 1768.544372] Key type .llcrypt registered [ 1768.777505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1768.791915] alg: No test for adler32 (adler32-zlib) [ 1769.922518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1769.981832] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1779.483994] Lustre: DEBUG MARKER: Initial discovery [ 1779.657592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.125@tcp [ 1782.034333] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1782.328982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1782.378719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.125@tcp [ 1782.426456] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1782.449973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1782.499460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.125@tcp -r 1 -m GET -l 3 [ 1782.527959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.125@tcp -r 1 -m GET -e local_error [ 1782.603594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.125@tcp -r 1 -m PUT -l 6 [ 1783.332843] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 1784.224996] Lustre: DEBUG MARKER: Force oleg125-client.virtnet to discover 192.168.201.125@tcp (in background) [ 1784.546987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.125@tcp [ 1785.814280] Lustre: DEBUG MARKER: Force oleg125-server to discover 192.168.201.25@tcp [ 1787.615366] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1788.496170] Lustre: DEBUG MARKER: Wait for 70486 [ 1790.134797] Lustre: DEBUG MARKER: Finished wait on 70486 [ 1790.495600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1790.602551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1790.606458] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1791.011471] LNet: 70935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1791.018132] LNetError: 70935:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1791.036295] LNet: Removed LNI 192.168.201.25@tcp [ 1791.468535] Key type .llcrypt unregistered [ 1791.471609] Key type ._llcrypt unregistered [ 1809.101043] Key type ._llcrypt registered [ 1809.104025] Key type .llcrypt registered [ 1810.485648] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 04:40:17 (1769074817) [ 1811.222355] Key type .llcrypt unregistered [ 1811.226984] Key type ._llcrypt unregistered [ 1818.884635] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1819.137087] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1819.148672] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1819.166266] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1821.737703] Key type ._llcrypt registered [ 1821.739424] Key type .llcrypt registered [ 1821.917645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1821.965777] alg: No test for adler32 (adler32-zlib) [ 1822.884736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1822.956840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1822.976396] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1823.672915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1823.696407] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1831.832880] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 04:40:39 (1769074839) [ 1832.031747] LNet: 73008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1832.037709] LNetError: 73008:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1832.041040] LNet: Removed LNI 192.168.201.25@tcp [ 1832.099823] LNet: Removed LNI 10.1.2.3@tcp [ 1832.985241] Key type .llcrypt unregistered [ 1832.991075] Key type ._llcrypt unregistered [ 1840.468271] Key type ._llcrypt registered [ 1840.470143] Key type .llcrypt registered [ 1840.797407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1840.808737] alg: No test for adler32 (adler32-zlib) [ 1841.810493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.870168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1841.922096] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1842.534176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1842.560589] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1843.596742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.25@tcp [ 1844.640137] LNetError: Unexpected error -22 connecting to 192.168.201.25@tcp at host 192.168.201.25:988 [ 1851.388875] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 04:40:58 (1769074858) [ 1851.704655] LNet: 74138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.717069] LNetError: 74138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.742842] LNet: Removed LNI 192.168.201.25@tcp [ 1852.786786] LNet: Removed LNI 10.1.2.3@tcp [ 1853.721262] Key type .llcrypt unregistered [ 1853.723516] Key type ._llcrypt unregistered [ 1860.310401] Key type ._llcrypt registered [ 1860.312057] Key type .llcrypt registered [ 1860.661533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1860.722977] alg: No test for adler32 (adler32-zlib) [ 1861.651462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1861.734716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1861.780568] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1862.549813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1862.567103] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1863.438716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.25@tcp1 --nid 192.168.201.25@tcp2 [ 1867.610893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp1 192.168.201.25@tcp2 [ 1867.658194] LNetError: 74970:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.25@tcp1 to 192.168.201.25@tcp2 - Route table may be misconfigured [ 1867.729491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp2 192.168.201.25@tcp1 [ 1875.409789] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 04:41:22 (1769074882) [ 1875.896915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1875.912595] LNet: 75544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1875.922364] LNetError: 75544:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1875.937704] LNet: Removed LNI 192.168.201.25@tcp1 [ 1876.969344] LNet: Removed LNI 192.168.201.25@tcp2 [ 1877.035587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1877.089683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1877.116428] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1877.779259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1877.798039] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1878.652594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1878.700582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp 192.168.201.25@tcp [ 1878.750580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp1 192.168.201.25@tcp1 [ 1878.815062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.25@tcp -d 192.168.201.25@tcp -e network_timeout [ 1878.914579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -e network_timeout [ 1879.010392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -e network_timeout [ 1879.085337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -e network_timeout [ 1879.169941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1879.208040] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1879.228296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1879.314579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1879.482617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1880.159233] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 1886.059461] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 04:41:33 (1769074893) [ 1886.233523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1886.278127] LNet: 76406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1886.289511] LNetError: 76406:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1886.301192] LNetError: 76406:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1886.311182] LNet: Removed LNI 192.168.201.25@tcp [ 1887.334864] LNet: Removed LNI 192.168.201.25@tcp1 [ 1887.371311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1887.462955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1887.705263] LNet: 76442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1887.998764] Key type .llcrypt unregistered [ 1888.000908] Key type ._llcrypt unregistered [ 1899.999453] Key type ._llcrypt registered [ 1900.001811] Key type .llcrypt registered [ 1901.541751] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 04:41:48 (1769074908) [ 1902.110983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1902.125539] alg: No test for adler32 (adler32-zlib) [ 1903.158098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.025656] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1906.602701] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 04:41:54 (1769074914) [ 1906.958257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1906.985988] LNet: 77537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1907.014794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1907.070775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1907.108107] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1907.849227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1907.869628] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1908.950434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1908.999990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp1 [ 1909.050476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp1 [ 1918.054710] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 04:42:04 (1769074924) [ 1919.736932] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg125-server [ 1921.444898] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 04:42:09 (1769074929) [ 1923.554436] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg125-server [ 1925.720382] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 04:42:12 (1769074932) [ 1928.190529] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg125-server [ 1930.299790] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 04:42:17 (1769074937) [ 1932.573244] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg125-server [ 1934.832340] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 04:42:21 (1769074941) [ 1937.123656] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg125-server [ 1939.879572] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 04:42:26 (1769074946) [ 1941.873569] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg125-server [ 1943.801315] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 04:42:31 (1769074951) [ 1945.792245] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg125-server [ 1947.854404] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 04:42:35 (1769074955) [ 1949.612412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg125-server [ 1951.191972] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 04:42:38 (1769074958) [ 1953.113542] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg125-server [ 1954.688896] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 04:42:42 (1769074962) [ 1955.026041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1955.049414] LNet: 81415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1955.061239] LNetError: 81415:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1955.073469] LNet: Removed LNI 192.168.201.25@tcp [ 1956.134850] LNet: Removed LNI 192.168.201.25@tcp1 [ 1956.184919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1956.271989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1956.321726] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1957.130395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1957.225062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1957.447204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1957.485555] LNet: 81563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1957.491882] LNetError: 81563:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1957.499784] LNetError: 81563:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1957.512284] LNet: Removed LNI 192.168.201.25@tcp [ 1957.525925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1957.566393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1957.605087] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1958.454437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1958.684557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1958.714420] LNetError: 81708:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.734057] LNet: Removed LNI 192.168.201.25@tcp [ 1958.767924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.845648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1958.864470] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1959.639302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1959.754961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1959.976846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1960.002204] LNet: 81855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.009796] LNet: 81855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1961.075475] LNet: Removed LNI 192.168.201.25@tcp [ 1961.127819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1961.238213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1961.278328] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1962.255477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1962.488948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1962.508254] LNetError: 82001:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1962.512811] LNetError: 82001:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1962.520774] LNet: Removed LNI 192.168.201.25@tcp [ 1962.532847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.576792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1962.592139] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1963.117905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1963.195679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1963.425446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.518624] LNet: Removed LNI 192.168.201.25@tcp [ 1964.567963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1964.665917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1964.712797] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1965.624434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1965.848491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.864674] LNet: 82293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1965.870316] LNet: 82293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1966.894801] LNet: Removed LNI 192.168.201.25@tcp [ 1966.950467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1967.030114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1967.061403] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1967.783285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1967.882528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1969.279109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1969.317015] LNetError: 82450:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1969.322155] LNetError: 82450:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1970.348747] LNet: Removed LNI 192.168.201.25@tcp [ 1970.389614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1970.477481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1970.521161] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1971.271301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1971.494910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1972.582449] LNet: Removed LNI 192.168.201.25@tcp [ 1972.603903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1972.697850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1972.717932] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1973.330978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1973.420435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1973.696960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1974.778592] LNet: Removed LNI 192.168.201.25@tcp [ 1974.827659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1974.924928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1974.989726] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1975.936650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1976.140429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1976.158902] LNet: 82888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.164909] LNet: 82888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1977.200540] LNet: Removed LNI 192.168.201.25@tcp [ 1977.235749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.315664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1977.343260] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1978.132789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1978.219964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1978.434732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1978.452397] LNetError: 83035:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1978.460438] LNetError: 83035:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1978.472878] LNet: Removed LNI 192.168.201.25@tcp [ 1978.489495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1978.533775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1978.548105] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1979.185521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1979.368070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1980.455599] LNet: Removed LNI 192.168.201.25@tcp [ 1980.495610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1980.547670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1980.582929] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1981.416802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1981.471175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1981.496577] LNet: Removed LNI 192.168.201.25@tcp [ 1981.511840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.597620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1981.626525] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1982.552803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1989.757317] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 04:43:17 (1769074997) [ 1989.994957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1990.024380] LNet: Removed LNI 192.168.201.25@tcp [ 1990.043046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1990.091649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1990.122091] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/180] [ 1991.239958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1991.266349] LNet: Removed LNI 192.168.201.25@tcp231 [ 1991.275859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1991.342410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1991.361393] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/180] [ 1991.365320] LNet: 84163:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1992.009399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1992.068286] LNet: Removed LNI 192.168.201.25@tcp231 [ 1992.084165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1992.121075] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/180] [ 1993.379596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1993.410538] LNet: 84307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1993.417868] LNet: 84307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1993.432980] LNet: Removed LNI 192.168.201.25@tcp231 [ 1993.451465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.558647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1993.609332] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/0] [ 1993.616507] LNet: 84315:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1994.300686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1994.334919] LNet: Removed LNI 192.168.201.25@tcp231 [ 1994.351665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1994.410784] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/0] [ 1995.788450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1995.837780] LNetError: 84459:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1995.842286] LNetError: 84459:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1995.864751] LNet: Removed LNI 192.168.201.25@tcp231 [ 1995.898791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1996.025334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1996.067759] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/60] [ 1996.083457] LNet: 84467:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1996.488548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1996.524633] LNet: Removed LNI 192.168.201.25@tcp231 [ 1996.542157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1996.568916] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/60] [ 1997.668690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1997.696134] LNet: Removed LNI 192.168.201.25@tcp231 [ 1997.728197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1997.809443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1997.838970] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/180] [ 1998.453529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1998.496397] LNet: Removed LNI 192.168.201.25@tcp231 [ 1998.515288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1998.548365] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/180] [ 2007.829546] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 04:43:35 (1769075015) [ 2008.034599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2008.097407] LNet: Removed LNI 192.168.201.25@tcp231 [ 2008.129405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2008.202457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 2008.238632] LNet: Added LNI 192.168.201.25@tcp232 [8/256/0/180] [ 2009.071297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2009.116992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2009.163090] LNet: Removed LNI 192.168.201.25@tcp232 [ 2009.188694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 2009.208418] LNet: Added LNI 192.168.201.25@tcp232 [8/256/0/180] [ 2009.852370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2009.888270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2009.918281] LNet: Removed LNI 192.168.201.25@tcp232 [ 2009.942729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 2009.967450] LNet: Added LNI 192.168.201.25@tcp232 [8/256/0/180] [ 2010.672545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2010.723948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2010.752863] LNet: Removed LNI 192.168.201.25@tcp232 [ 2010.767827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 2010.789541] LNet: Added LNI 192.168.201.25@tcp232 [8/256/0/180] [ 2011.392449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2011.420765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2011.452983] LNet: Removed LNI 192.168.201.25@tcp232 [ 2011.463381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 2018.322429] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 04:43:45 (1769075025) [ 2019.558251] Key type .llcrypt unregistered [ 2019.561220] Key type ._llcrypt unregistered [ 2026.159673] Key type ._llcrypt registered [ 2026.165095] Key type .llcrypt registered [ 2026.444709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2026.497344] alg: No test for adler32 (adler32-zlib) [ 2027.446786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2027.480314] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2034.982706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 2043.817751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2043.884406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2043.934896] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 2046.431050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_interrupt -r 2 [ 2046.527339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_interrupt -r 2 [ 2046.626710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_interrupt -r 2 [ 2046.682873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_interrupt -r 2 [ 2046.742680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_interrupt -r 2 [ 2046.793629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_interrupt -r 2 [ 2046.883514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_interrupt -r 2 [ 2046.965794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_interrupt -r 2 [ 2047.028548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp 192.168.201.125@tcp [ 2047.062506] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 2047.070562] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 2047.416563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2047.463860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2047.506875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2047.553389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2047.606571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2047.653766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2047.696495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2047.734964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2047.780402] LNet: 87114:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2047.800548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2047.837832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2047.882846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2047.925820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2047.964574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2048.001434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2048.705532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_dropped -r 2 [ 2048.754905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_dropped -r 2 [ 2048.796565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_dropped -r 2 [ 2048.838935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_dropped -r 2 [ 2048.879872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_dropped -r 2 [ 2048.919933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_dropped -r 2 [ 2048.961751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_dropped -r 2 [ 2049.064687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_dropped -r 2 [ 2049.116671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp 192.168.201.125@tcp [ 2049.148717] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 2049.482760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2049.513684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2049.556383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2049.588290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2049.611186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2049.632916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2049.660704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2049.716340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2049.734146] LNet: 87234:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2049.759681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2049.787753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2049.838319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2049.865919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2049.906657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2049.956104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2050.562245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_aborted -r 2 [ 2050.600717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_aborted -r 2 [ 2050.631263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_aborted -r 2 [ 2050.659414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_aborted -r 2 [ 2050.690700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_aborted -r 2 [ 2050.722361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_aborted -r 2 [ 2050.746429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_aborted -r 2 [ 2050.780219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_aborted -r 2 [ 2050.818211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp 192.168.201.125@tcp [ 2050.836577] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 2051.063417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2051.092594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2051.122481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2051.145438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2051.175371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2051.203827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2051.250072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2051.317068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2051.348432] LNet: 87355:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2051.374560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2051.409642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2051.435343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2051.473681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2051.519825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2051.554396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2052.072310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_no_route -r 2 [ 2052.127936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_no_route -r 2 [ 2052.196479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_no_route -r 2 [ 2052.260882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_no_route -r 2 [ 2052.325840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_no_route -r 2 [ 2052.373261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_no_route -r 2 [ 2052.412258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_no_route -r 2 [ 2052.447342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_no_route -r 2 [ 2052.486226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp 192.168.201.125@tcp [ 2052.868362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2052.938526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2052.989361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2053.015446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2053.046393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2053.079926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2053.110842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2053.138384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2053.169440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2053.195203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2053.238699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2053.286643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2053.370143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2053.413836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2054.222824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_timeout -r 2 [ 2054.263928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_timeout -r 2 [ 2054.298411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_timeout -r 2 [ 2054.326460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_timeout -r 2 [ 2054.355515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_timeout -r 2 [ 2054.399187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_timeout -r 2 [ 2054.435332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_timeout -r 2 [ 2054.480652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_timeout -r 2 [ 2054.513574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp 192.168.201.125@tcp [ 2054.529617] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 2054.534475] LNet: Skipped 1 previous similar message [ 2054.971357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2055.030434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2055.065697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2055.154788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2055.220650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2055.297332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2055.352502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2055.392724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2055.421793] LNet: 87595:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2055.433304] LNet: 87595:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 2055.464676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2055.515287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2055.569576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2055.643722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2055.727834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2055.788704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2056.499233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e remote_dropped -r 2 [ 2056.562860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e remote_dropped -r 2 [ 2056.626523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e remote_dropped -r 2 [ 2056.695785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e remote_dropped -r 2 [ 2056.770729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e remote_dropped -r 2 [ 2056.837581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e remote_dropped -r 2 [ 2056.908382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e remote_dropped -r 2 [ 2056.979866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e remote_dropped -r 2 [ 2057.041013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp 192.168.201.125@tcp [ 2057.128761] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.125@tcp [ 2057.495429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2057.566642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2057.614953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2057.661784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2057.720792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2057.780874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2057.855721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2057.913703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2057.988820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2058.037789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2058.090677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2058.141176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2058.190781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2058.250326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2065.002928] LNet: 87789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2065.014676] LNetError: 87789:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2065.037228] LNet: Removed LNI 192.168.201.25@tcp [ 2065.053348] LNet: Removed LNI 192.168.201.25@tcp1 [ 2065.489624] Key type .llcrypt unregistered [ 2065.491650] Key type ._llcrypt unregistered [ 2082.098989] Key type ._llcrypt registered [ 2082.100892] Key type .llcrypt registered [ 2083.246944] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 04:44:50 (1769075090) [ 2083.838295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2083.881841] alg: No test for adler32 (adler32-zlib) [ 2084.824651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.875228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2084.896305] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2085.601397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2085.625720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.25@tcp [ 2085.641932] LNetError: 88763:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.201.25@tcp to recovery [ 2085.720817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.25@tcp [ 2085.861975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2092.812243] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 04:44:59 (1769075099) [ 2093.024411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2093.044609] LNet: 89348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2093.050563] LNetError: 89348:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2093.074486] LNet: Removed LNI 192.168.201.25@tcp [ 2093.103094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2093.168545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2093.207197] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2093.847903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2093.880370] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 2095.028113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.25@tcp [ 2095.057406] LNetError: 89619:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.201.25@tcp to recovery [ 2095.200174] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 2095.205890] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 2096.323738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.25@tcp [ 2103.955472] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 04:45:11 (1769075111) [ 2106.179239] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg125-server [ 2108.765680] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 04:45:15 (1769075115) [ 2109.223034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2109.269912] LNet: 90567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.279840] LNetError: 90567:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2109.288845] LNet: Removed LNI 192.168.201.25@tcp [ 2109.311439] LNet: Removed LNI 192.168.201.25@tcp1 [ 2109.357469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2109.453470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2109.485639] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2110.522550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2110.608213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2110.658372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2110.772800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2110.831940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2110.890683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2118.210357] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 04:45:25 (1769075125) [ 2118.508574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2118.559099] LNet: 91293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2118.566868] LNetError: 91293:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2118.575591] LNetError: 91293:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2118.603238] LNet: Removed LNI 192.168.201.25@tcp [ 2118.641673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2118.710847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2118.729872] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2127.316949] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 04:45:34 (1769075134) [ 2127.592277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.630735] LNet: 91997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2127.637732] LNetError: 91997:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2127.658930] LNet: Removed LNI 192.168.201.25@tcp [ 2127.688360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.747207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2127.784704] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2135.534705] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 04:45:42 (1769075142) [ 2137.378091] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2139.986678] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 04:45:46 (1769075146) [ 2140.697502] LNet: 93032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2140.706896] LNetError: 93032:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2140.736674] LNet: Removed LNI 192.168.201.25@tcp [ 2141.624396] Key type .llcrypt unregistered [ 2141.626746] Key type ._llcrypt unregistered [ 2149.280384] Key type ._llcrypt registered [ 2149.282669] Key type .llcrypt registered [ 2149.545771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2149.561601] alg: No test for adler32 (adler32-zlib) [ 2150.520252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2150.542310] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2159.721740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 2166.209066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.125@tcp [ 2166.784778] LNet: 93446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2166.804149] LNetError: 93446:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2166.821248] LNet: Removed LNI 192.168.201.25@tcp [ 2167.548785] Key type .llcrypt unregistered [ 2167.554084] Key type ._llcrypt unregistered [ 2180.523234] Key type ._llcrypt registered [ 2180.524937] Key type .llcrypt registered [ 2181.751369] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2183.520684] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2185.137647] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 04:46:32 (1769075192) [ 2185.775179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2185.793939] alg: No test for adler32 (adler32-zlib) [ 2186.699906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.811440] LNet: 94532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2187.419541] Key type .llcrypt unregistered [ 2187.424099] Key type ._llcrypt unregistered [ 2195.716714] Key type ._llcrypt registered [ 2195.728190] Key type .llcrypt registered [ 2196.233935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2196.280137] alg: No test for adler32 (adler32-zlib) [ 2197.506148] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2197.517066] LNet: 94785:0:(config.c:1226:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 2197.543804] LNet: 94785:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2197.595530] LNetError: 94788:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.201.27@tcp has gone from up to down [ 2197.658837] LNetError: 94797:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.28@tcp is being used as a gateway but routing feature is not turned on [ 2198.557323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2198.585980] LNet: 94843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2198.592884] LNetError: 94843:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2198.622646] LNet: Removed LNI 192.168.201.25@tcp [ 2198.705223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2198.809891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2198.845102] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2198.853060] LNet: 94850:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2198.861534] LNet: 94850:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 2198.867443] LNetError: 94850:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.201.27@tcp is a gateway. Can not delete it [ 2198.874864] LNetError: 94850:0:(peer.c:2217:lnet_del_peer_ni()) 192.168.201.27@tcp is a router. Can not be deleted [ 2198.891268] LNet: 94850:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2198.895162] LNetError: 94847:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.28@tcp is being used as a gateway but routing feature is not turned on [ 2198.913112] LNetError: 94847:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 2198.925710] LNetError: 94847:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.201.28@tcp has gone from up to down [ 2198.937625] LNetError: 94847:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 2208.732539] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 04:46:55 (1769075215) [ 2211.432338] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg125-server [ 2213.856560] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 04:47:00 (1769075220) [ 2216.545132] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg125-server [ 2218.560042] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 04:47:05 (1769075225) [ 2220.392658] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 2222.209036] Key type lgssc registered [ 2223.417995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2244.663928] Key type lgssc unregistered [ 2245.044052] LNet: 97270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2245.052857] LNetError: 97270:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2245.078343] LNet: Removed LNI 192.168.201.25@tcp [ 2245.844770] Key type .llcrypt unregistered [ 2245.847671] Key type ._llcrypt unregistered [ 2261.692111] Key type ._llcrypt registered [ 2261.694140] Key type .llcrypt registered [ 2262.940787] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 04:47:50 (1769075270) [ 2263.417750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2263.440513] alg: No test for adler32 (adler32-zlib) [ 2264.411311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2264.472687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2264.507377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2264.549688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2264.604899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2271.930347] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 04:47:58 (1769075278) [ 2272.215902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2272.249015] LNet: 98671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.307933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.407337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2272.443653] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2273.645695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2273.669935] LNet: 98818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.678938] LNetError: 98818:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.692197] LNet: Removed LNI 192.168.201.25@tcp [ 2273.734340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.783350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2273.805734] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2274.553482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2274.576244] LNetError: 98846:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2274.591870] LNet: Removed LNI 192.168.201.25@tcp [ 2274.644130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2274.726042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2274.753661] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2282.773449] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 04:48:10 (1769075290) [ 2283.080399] LNet: 99436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2283.084520] LNet: 99436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2283.106047] LNetError: 99436:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2283.130164] LNet: Removed LNI 192.168.201.25@tcp [ 2283.873687] Key type .llcrypt unregistered [ 2283.879501] Key type ._llcrypt unregistered [ 2291.714089] Key type ._llcrypt registered [ 2291.717614] Key type .llcrypt registered [ 2292.157284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2292.184591] alg: No test for adler32 (adler32-zlib) [ 2293.215221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.292457] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2293.294742] LNet: *** cfs_fail_loc=e003, val=100*** [ 2294.077930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 2294.119428] LNet: *** cfs_fail_loc=e003, val=100*** [ 2294.127601] LNet: Skipped 2 previous similar messages [ 2294.144546] Ping reply received with mapping: [ 2294.148871] found metadata reply NI[192.168.201.25@tcp] [ 2294.153488] 19 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2294.157692] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2294.162582] 5a 5a 5a 5a 5a 5a 5a 5a [ 2294.409282] LNet: 99831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2294.426512] LNetError: 99831:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2294.445915] LNet: Removed LNI 192.168.201.25@tcp [ 2295.035982] Key type .llcrypt unregistered [ 2295.039242] Key type ._llcrypt unregistered [ 2308.453688] Key type ._llcrypt registered [ 2308.460498] Key type .llcrypt registered [ 2310.095333] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 04:48:37 (1769075317) [ 2311.037574] Key type .llcrypt unregistered [ 2311.040494] Key type ._llcrypt unregistered [ 2318.803751] Key type ._llcrypt registered [ 2318.809655] Key type .llcrypt registered [ 2319.166976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2319.197549] alg: No test for adler32 (adler32-zlib) [ 2320.379751] LNetError: 100866:0:(api-ni.c:2732:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2320.898155] Key type .llcrypt unregistered [ 2320.899808] Key type ._llcrypt unregistered [ 2327.253447] Key type ._llcrypt registered [ 2327.257171] Key type .llcrypt registered [ 2328.692513] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 04:48:55 (1769075335) [ 2329.331383] Key type .llcrypt unregistered [ 2329.333067] Key type ._llcrypt unregistered [ 2336.195507] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2336.210720] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2338.889419] Key type ._llcrypt registered [ 2338.892065] Key type .llcrypt registered [ 2339.095333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2339.113659] alg: No test for adler32 (adler32-zlib) [ 2340.096804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.152876] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2351.151803] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 04:49:18 (1769075358) [ 2351.325958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2351.340352] LNet: 102634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2351.346457] LNetError: 102634:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2351.366521] LNet: Removed LNI 10.1.2.3@tcp [ 2351.397154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2352.505712] Key type .llcrypt unregistered [ 2352.508265] Key type ._llcrypt unregistered [ 2363.466396] Key type ._llcrypt registered [ 2363.471789] Key type .llcrypt registered [ 2363.786317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2363.830374] alg: No test for adler32 (adler32-zlib) [ 2364.813081] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2364.905679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 2365.073919] LNet: 102925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2365.080848] LNetError: 102925:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2366.118283] LNet: Removed LNI 192.168.201.25@tcp [ 2366.898481] Key type .llcrypt unregistered [ 2366.903212] Key type ._llcrypt unregistered [ 2381.694530] Key type ._llcrypt registered [ 2381.697296] Key type .llcrypt registered [ 2383.419969] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 04:49:50 (1769075390) [ 2384.225654] Key type .llcrypt unregistered [ 2384.230912] Key type ._llcrypt unregistered [ 2392.150946] Key type ._llcrypt registered [ 2392.153249] Key type .llcrypt registered [ 2392.458248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2392.472382] alg: No test for adler32 (adler32-zlib) [ 2401.196212] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 04:50:08 (1769075408) [ 2401.530263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2401.567902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2401.624036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2401.647155] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2402.379795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2402.412576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2402.453588] LNet: Removed LNI 192.168.201.25@tcp [ 2402.773630] LNet: 104664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2403.331388] Key type .llcrypt unregistered [ 2403.333421] Key type ._llcrypt unregistered [ 2416.128792] Key type ._llcrypt registered [ 2416.131722] Key type .llcrypt registered [ 2418.527148] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 04:50:24 (1769075424) [ 2419.078557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2419.103264] alg: No test for adler32 (adler32-zlib) [ 2420.142512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2420.214725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2420.239175] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2421.333573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 2421.379461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.25@tcp [ 2421.398060] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2421.404641] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2421.410987] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2421, 0, 0 [ 2421.415882] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2421.422087] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2421.426611] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2421.428563] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2421, 0, 0 [ 2421.431920] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2421.434327] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2421.436647] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2421.439699] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2421, 0, 2471 [ 2421.442464] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2421.444339] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2421.448571] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2421.452484] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2421, 0, 2471 [ 2421.460838] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2429.449797] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 04:50:36 (1769075436) [ 2430.151679] LNet: 106202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2430.168390] LNetError: 106202:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2431.207732] LNet: Removed LNI 192.168.201.25@tcp [ 2432.128482] Key type .llcrypt unregistered [ 2432.130437] Key type ._llcrypt unregistered [ 2440.194925] Key type ._llcrypt registered [ 2440.201233] Key type .llcrypt registered [ 2440.539563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2440.602058] alg: No test for adler32 (adler32-zlib) [ 2441.710373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2441.778944] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2451.166680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 2461.796842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2461.853525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2461.882690] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 2463.285077] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2463.422919] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2463.426222] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2463.432230] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2465.848240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2465.865104] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2478.898568] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 04:51:25 (1769075485) [ 2479.166487] LNet: 107700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2479.179101] LNetError: 107700:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2479.213093] LNet: Removed LNI 192.168.201.25@tcp [ 2479.218517] LNet: Removed LNI 192.168.201.25@tcp1 [ 2479.270438] LNet: Removed LNI 10.1.2.3@tcp99 [ 2480.193256] Key type .llcrypt unregistered [ 2480.197386] Key type ._llcrypt unregistered [ 2489.703894] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2489.714825] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2491.368495] Key type ._llcrypt registered [ 2491.373240] Key type .llcrypt registered [ 2491.698747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2491.722509] alg: No test for adler32 (adler32-zlib) [ 2492.738627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2492.824620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2492.864233] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2493.831735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2493.870121] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2496.736980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2496.782607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2496.825300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2496.853645] LNetError: 108441:0:(peer.c:2225:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2496.912235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2496.978356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2497.629050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2497.748966] LNet: 108461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2497.756637] LNetError: 108461:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2497.766374] LNet: Removed LNI 192.168.201.25@tcp [ 2497.783759] LNet: Removed LNI 10.1.2.3@tcp [ 2498.687453] Key type .llcrypt unregistered [ 2498.690396] Key type ._llcrypt unregistered [ 2514.118075] Key type ._llcrypt registered [ 2514.119435] Key type .llcrypt registered [ 2516.064305] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 04:52:02 (1769075522) [ 2516.682627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2516.698648] alg: No test for adler32 (adler32-zlib) [ 2517.784554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2517.873318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2517.944947] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2518.799327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 2518.909962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg125-client@tcp [ 2526.531338] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 04:52:13 (1769075533) [ 2526.913553] LNet: 109994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2526.922749] LNetError: 109994:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2526.939564] LNet: Removed LNI 192.168.201.25@tcp [ 2527.507494] Key type .llcrypt unregistered [ 2527.509127] Key type ._llcrypt unregistered [ 2535.457072] Key type ._llcrypt registered [ 2535.458966] Key type .llcrypt registered [ 2535.626663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2535.659956] alg: No test for adler32 (adler32-zlib) [ 2536.542593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2536.570108] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2536.615506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2543.980813] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 04:52:31 (1769075551) [ 2545.946571] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2548.242328] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 04:52:35 (1769075555) [ 2549.741853] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2552.049810] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 04:52:39 (1769075559) [ 2552.167277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2552.213266] LNet: 111404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2552.237032] LNetError: 111404:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2552.297329] LNet: Removed LNI 192.168.201.25@tcp [ 2552.332884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2552.441257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2552.457711] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2553.112776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2553.174563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2561.019911] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 04:52:48 (1769075568) [ 2561.472246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2561.504286] LNet: 112121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2561.511200] LNetError: 112121:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2561.530689] LNet: Removed LNI 192.168.201.25@tcp [ 2561.554725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2561.590345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2561.606882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2562.474486] Key type .llcrypt unregistered [ 2562.477344] Key type ._llcrypt unregistered [ 2575.667740] Key type ._llcrypt registered [ 2575.669553] Key type .llcrypt registered [ 2577.129671] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 04:53:04 (1769075584) [ 2577.602976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2577.613381] alg: No test for adler32 (adler32-zlib) [ 2578.655415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2578.738648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2578.768176] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2579.549916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2579.614865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2588.120739] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 04:53:14 (1769075594) [ 2588.419316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2588.454390] LNet: 113673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2588.460642] LNetError: 113673:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2588.488898] LNet: Removed LNI 192.168.201.25@tcp [ 2588.512650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2588.599955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2588.656368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2596.472898] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 04:53:23 (1769075603) [ 2596.812064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2596.843397] LNet: 114254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2596.873585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2596.952487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2597.561606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2597.605954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2597.714078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2598.362561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2598.405964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2598.544530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2598.558619] LNet: 114287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2598.583267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2598.638579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2599.117969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2599.192208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2599.248804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2599.708887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2599.770460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2607.903697] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 04:53:35 (1769075615) [ 2608.332323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2608.382662] LNet: 114887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2608.430792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2608.522654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2608.570574] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2609.458368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2609.482429] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 2610.331282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2610.431540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2610.816457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2610.858000] LNetError: 115167:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2610.864038] LNet: Removed LNI 192.168.201.25@tcp [ 2610.896781] LNet: Removed LNI 192.168.201.25@tcp2 [ 2610.936185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2611.031966] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2611.051140] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 2611.059871] LNet: 115173:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2611.825437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2612.112574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2612.152121] LNetError: 115210:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2612.167534] LNetError: 115210:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2612.180452] LNet: Removed LNI 192.168.201.25@tcp [ 2612.197982] LNet: Removed LNI 192.168.201.25@tcp2 [ 2612.236217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2612.317159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2612.375834] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2613.284705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2613.314984] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 2614.662923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2614.732935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2614.869326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2615.324278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2615.356398] LNet: 115495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2615.362414] LNet: 115495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2615.368982] LNetError: 115495:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2615.374417] LNetError: 115495:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2615.379213] LNet: Removed LNI 192.168.201.25@tcp [ 2616.438685] LNet: Removed LNI 192.168.201.25@tcp2 [ 2616.526656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2616.639203] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2616.650366] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 2616.662359] LNet: 115501:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2617.595849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2617.932497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2617.962695] LNet: Removed LNI 192.168.201.25@tcp [ 2618.986727] LNet: Removed LNI 192.168.201.25@tcp2 [ 2619.058871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2619.209208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2619.298850] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2620.182883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2620.213241] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 2621.403248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2621.471413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2621.727461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2621.747283] LNetError: 115818:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2621.752988] LNetError: 115818:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2621.758742] LNet: Removed LNI 192.168.201.25@tcp [ 2621.769494] LNet: Removed LNI 192.168.201.25@tcp2 [ 2621.795776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2621.917685] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2621.925776] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 2621.931416] LNet: 115824:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2622.799141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2623.081314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2624.162105] LNet: Removed LNI 192.168.201.25@tcp [ 2624.180331] LNet: Removed LNI 192.168.201.25@tcp2 [ 2624.295624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2624.413079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2624.449774] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2625.604856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2625.640922] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 2626.727824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2626.809460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2626.891388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2627.129448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2627.144898] LNet: 116146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2627.150921] LNet: 116146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2628.191423] LNet: Removed LNI 192.168.201.25@tcp [ 2629.231458] LNet: Removed LNI 192.168.201.25@tcp2 [ 2629.262391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2629.322124] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2629.329455] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 2629.337321] LNet: 116152:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2629.981761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2630.235140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2630.249370] LNetError: 116189:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2630.253351] LNetError: 116189:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2630.260971] LNet: Removed LNI 192.168.201.25@tcp [ 2631.334197] LNet: Removed LNI 192.168.201.25@tcp2 [ 2631.424624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2631.529709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2631.575768] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2632.403246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2632.424627] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 2633.691447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2633.716953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2633.778621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2634.348897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2635.423215] LNet: Removed LNI 192.168.201.25@tcp [ 2635.438966] LNet: Removed LNI 192.168.201.25@tcp2 [ 2635.507957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2635.641152] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2635.652196] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 2635.672871] LNet: 116492:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2636.356489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2636.756148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2637.791510] LNet: Removed LNI 192.168.201.25@tcp [ 2637.800928] LNet: Removed LNI 192.168.201.25@tcp2 [ 2637.865211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2637.961459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2638.010488] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2638.980415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2639.005926] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 2640.043720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2640.106386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2640.147925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2640.237264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2640.801833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2641.887244] LNet: Removed LNI 192.168.201.25@tcp [ 2642.920341] LNet: Removed LNI 192.168.201.25@tcp2 [ 2642.998124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2643.078407] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2643.083667] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 2643.953799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2653.779644] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2655.588085] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 04:54:22 (1769075662) [ 2655.947429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2655.966301] LNet: 117612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2655.977171] LNet: 117612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2655.984500] LNetError: 117612:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2655.989653] LNetError: 117612:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2655.996414] LNet: Removed LNI 192.168.201.25@tcp [ 2657.063252] LNet: Removed LNI 192.168.201.25@tcp2 [ 2657.106215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2657.492352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2657.558066] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2658.848668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2658.875554] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2660.099116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2660.133101] LNet: Removed LNI 192.168.201.25@tcp [ 2660.141532] LNet: Removed LNI 10.1.2.3@tcp [ 2660.164996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2660.229931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2660.259522] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2660.271745] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2661.045724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2661.089054] LNet: Removed LNI 192.168.201.25@tcp [ 2661.108065] LNet: Removed LNI 10.1.2.3@tcp [ 2669.304692] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 04:54:36 (1769075676) [ 2669.406911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2669.509792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2669.693922] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2669.904875] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2669.917221] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2669.925046] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2672.345578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2672.390122] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2673.232423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2673.268321] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2675.974775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 2676.088861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2676.176845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.25@tcp -m PUT -e local_timeout -r 1 [ 2676.254146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2676.338967] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2676.354329] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2676.657583] LNetError: 118555:0:(lib-move.c:4947:lnet_parse()) 192.168.201.25@tcp, src 192.168.201.25@tcp: Dropping PUT (error -108 looking up sender) [ 2678.687432] LNet: Removed LNI 192.168.201.25@tcp [ 2678.704327] LNet: Removed LNI 10.1.2.3@tcp [ 2679.277196] Key type .llcrypt unregistered [ 2679.281312] Key type ._llcrypt unregistered [ 2693.930948] Key type ._llcrypt registered [ 2693.941875] Key type .llcrypt registered [ 2695.898769] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 04:55:02 (1769075702) [ 2696.255939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2696.283236] alg: No test for adler32 (adler32-zlib) [ 2697.192159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2697.716281] LNet: 119862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2698.521610] Key type .llcrypt unregistered [ 2698.528151] Key type ._llcrypt unregistered [ 2705.699928] Key type ._llcrypt registered [ 2705.704465] Key type .llcrypt registered [ 2706.168448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2706.216481] alg: No test for adler32 (adler32-zlib) [ 2707.258412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2707.332906] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2716.929713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 2717.246698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2718.187771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.125@tcp [ 2718.544623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2718.627877] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.125@tcp [ 2718.631867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2718.703875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2718.786182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2718.914261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2719.010958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2719.149276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2719.283640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2719.340941] LNet: 120309:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2719.396246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2719.482796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2719.577308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2719.626583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2719.680717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2719.758575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2720.708392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2720.946994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2720.990936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2721.029841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2721.073750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2721.130519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2721.197087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2721.247225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2721.297942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2721.317450] LNet: 120426:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2721.339618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2721.371881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2721.403307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2721.430372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2721.456389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2721.486497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2722.393616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.125@tcp [ 2722.684133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2722.731435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2722.761378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2722.791478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2722.819789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2722.859452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2722.909796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2722.989257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2723.017331] LNet: 120543:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2723.040848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2723.084624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2723.143870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2723.194723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2723.290058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2723.334704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2724.034120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2724.288486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2724.318339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2724.372477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2724.425584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2724.477577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2724.542804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2724.599612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2724.652098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2724.718612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2724.762093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2724.815583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2724.847857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2724.879698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2724.924832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2725.308364] LNet: 120706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2725.325030] LNetError: 120706:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2725.366952] LNet: Removed LNI 192.168.201.25@tcp [ 2725.853862] Key type .llcrypt unregistered [ 2725.856821] Key type ._llcrypt unregistered [ 2747.669870] Key type ._llcrypt registered [ 2747.675722] Key type .llcrypt registered [ 2749.399215] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 04:55:56 (1769075756) [ 2750.327528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2750.365746] alg: No test for adler32 (adler32-zlib) [ 2751.396129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2752.131859] LNet: 121593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2753.020344] Key type .llcrypt unregistered [ 2753.021440] Key type ._llcrypt unregistered [ 2761.983907] Key type ._llcrypt registered [ 2761.992332] Key type .llcrypt registered [ 2762.407503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2762.438396] alg: No test for adler32 (adler32-zlib) [ 2763.458155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2763.531342] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2772.098444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 2782.615747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2782.696933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2782.743267] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 2784.223559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2785.165708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.125@tcp [ 2785.632518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2785.683894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2785.749757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2785.785088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2785.826833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2785.876496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2785.944723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2785.994814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2786.033482] LNet: 122244:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2786.073910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2786.079351] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.125@tcp [ 2786.134175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2786.214863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2786.264923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2786.336546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2786.426994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2787.443260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2787.797530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2787.835958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2787.864293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2787.894960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2787.931840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2787.980594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2788.023874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2788.064952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2788.087608] LNet: 122361:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2788.112649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2788.151979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2788.178572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2788.223764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2788.258926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2788.309345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2789.029573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.125@tcp [ 2789.315796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2789.351658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2789.391804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2789.417581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2789.496772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2789.547615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2789.617777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2789.672842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2789.701431] LNet: 122478:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2789.750712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2789.790107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2789.858987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2789.933980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2789.969647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2790.041476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2790.754800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2791.140925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2791.183371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2791.236232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2791.275928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2791.336710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2791.385770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2791.423564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2791.462092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2791.488251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2791.521409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2791.559445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2791.606919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2791.637845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2791.669837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2797.722369] LNet: 122676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2797.735980] LNetError: 122676:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2797.743340] LNet: Removed LNI 192.168.201.25@tcp [ 2797.779517] LNet: Removed LNI 192.168.201.25@tcp1 [ 2798.262952] Key type .llcrypt unregistered [ 2798.266501] Key type ._llcrypt unregistered [ 2816.920406] Key type ._llcrypt registered [ 2816.927579] Key type .llcrypt registered [ 2818.425164] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 04:57:05 (1769075825) [ 2827.106081] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 04:57:14 (1769075834) [ 2835.831924] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 04:57:22 (1769075842) [ 2844.442076] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 04:57:31 (1769075851) [ 2853.455566] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 04:57:40 (1769075860) [ 2863.338157] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 04:57:50 (1769075870) [ 2871.514541] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 04:57:58 (1769075878) [ 2882.040638] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 04:58:08 (1769075888) [ 2889.843691] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 04:58:17 (1769075897) [ 2898.497489] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 04:58:25 (1769075905) [ 2906.261566] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 04:58:33 (1769075913) [ 2914.428944] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 04:58:41 (1769075921) [ 2922.465601] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 04:58:49 (1769075929) [ 2930.406732] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 04:58:57 (1769075937) [ 2938.250369] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 04:59:05 (1769075945) [ 2947.284214] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 04:59:14 (1769075954) [ 2957.517575] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 04:59:24 (1769075964) [ 2966.744604] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 04:59:33 (1769075973) [ 2967.432627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2967.451988] alg: No test for adler32 (adler32-zlib) [ 2968.471036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2968.597913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2968.629290] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2970.188611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2970.220605] LNet: 133390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2970.235114] LNetError: 133390:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2970.268491] LNet: Removed LNI 192.168.201.25@tcp [ 2970.317417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2970.434700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2970.479838] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 2979.048900] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 04:59:46 (1769075986) [ 2987.541349] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 04:59:54 (1769075994) [ 2996.435463] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 05:00:03 (1769076003) [ 3005.219947] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 05:00:12 (1769076012) [ 3016.658055] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 05:00:23 (1769076023) [ 3025.416803] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 05:00:32 (1769076032) [ 3033.465867] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 05:00:40 (1769076040) [ 3040.440887] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 05:00:47 (1769076047) [ 3048.791146] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 05:00:55 (1769076055) [ 3057.515513] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 05:01:04 (1769076064) [ 3064.879615] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 05:01:12 (1769076072) [ 3072.405921] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 05:01:19 (1769076079) [ 3079.744849] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2798 sec ========== 05:01:27 (1769076087) [ 3080.251246] LNet: 140875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3080.257797] LNetError: 140875:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3080.297616] LNet: Removed LNI 192.168.201.25@tcp [ 3081.224923] Key type .llcrypt unregistered [ 3081.229592] Key type ._llcrypt unregistered [ 3095.741195] Key type ._llcrypt registered [ 3095.744151] Key type .llcrypt registered [ 3096.370219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3096.380845] alg: No test for adler32 (adler32-zlib) [ 3097.562959] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 3097.813644] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 3099.505592] Key type lgssc registered [ 3101.369949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3163.108711] Lustre: Mounted lustre-client [ 3168.843748] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3184.030711] LustreError: 143329:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9cc544b17000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 3184.061745] LustreError: 143329:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 3184.105190] Lustre: Unmounted lustre-client [ 3207.165994] Key type lgssc unregistered [ 3207.524852] LNet: 143811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3207.537802] LNetError: 143811:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3207.563871] LNet: Removed LNI 192.168.201.25@tcp [ 3208.518175] Key type .llcrypt unregistered [ 3208.521263] Key type ._llcrypt unregistered