[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 359194746 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001038] APIC: Switch to symmetric I/O mode setup [ 0.003485] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011022] pid_max: default: 32768 minimum: 301 [ 0.012226] LSM: Security Framework initializing [ 0.013048] Yama: becoming mindful. [ 0.014043] SELinux: Initializing. [ 0.015075] *** VALIDATE selinux *** [ 0.024301] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028783] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030013] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031090] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032127] *** VALIDATE tmpfs *** [ 0.033618] *** VALIDATE proc *** [ 0.034272] *** VALIDATE cgroup *** [ 0.035000] *** VALIDATE cgroup2 *** [ 0.036151] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037140] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039026] Spectre V2 : User space: Vulnerable [ 0.040005] Speculative Store Bypass: Vulnerable [ 0.042974] debug: unmapping init [mem 0xffffffff8dc59000-0xffffffff8dc60fff] [ 0.044850] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045464] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046011] ... version: 2 [ 0.046977] ... bit width: 48 [ 0.047006] ... generic registers: 4 [ 0.047964] ... value mask: 0000ffffffffffff [ 0.048007] ... max period: 00007fffffffffff [ 0.049006] ... fixed-purpose events: 3 [ 0.050006] ... event mask: 000000070000000f [ 0.051273] rcu: Hierarchical SRCU implementation. [ 0.053441] smp: Bringing up secondary CPUs ... [ 0.054452] x86: Booting SMP configuration: [ 0.055011] .... node #0, CPUs: #1 #2 #3 [ 0.058105] smp: Brought up 1 node, 4 CPUs [ 0.060009] smpboot: Max logical packages: 1 [ 0.061011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.245582] node 0 deferred pages initialised in 184ms [ 0.249128] devtmpfs: initialized [ 0.250227] x86/mm: Memory block size: 128MB [ 0.252396] gcov: version magic: 0x41383552 [ 0.255092] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.256059] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.257313] pinctrl core: initialized pinctrl subsystem [ 0.259137] [ 0.259589] ************************************************************* [ 0.261014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.263007] ** ** [ 0.264006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.265007] ** ** [ 0.267008] ** This means that this kernel is built to expose internal ** [ 0.268008] ** IOMMU data structures, which may compromise security on ** [ 0.270008] ** your system. ** [ 0.271006] ** ** [ 0.272006] ** If you see this message and you are not debugging the ** [ 0.274006] ** kernel, report this immediately to your vendor! ** [ 0.275004] ** ** [ 0.276005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.277005] ************************************************************* [ 0.279630] NET: Registered protocol family 16 [ 0.280305] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.282030] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.284035] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.287175] cpuidle: using governor menu [ 0.288789] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.291593] PCI: Using configuration type 1 for base access [ 0.294162] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.304103] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.305033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.309129] cryptd: max_cpu_qlen set to 1000 [ 0.312290] ACPI: Added _OSI(Module Device) [ 0.314012] ACPI: Added _OSI(Processor Device) [ 0.315011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.317014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.322315] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.332007] ACPI: Interpreter enabled [ 0.334126] ACPI: PM: (supports S0 S3 S4 S5) [ 0.336011] ACPI: Using IOAPIC for interrupt routing [ 0.339132] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.341432] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.351360] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.352024] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.354010] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.356068] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.359450] acpiphp: Slot [2] registered [ 0.361076] acpiphp: Slot [5] registered [ 0.362095] acpiphp: Slot [6] registered [ 0.363039] acpiphp: Slot [3] registered [ 0.363959] acpiphp: Slot [4] registered [ 0.365088] acpiphp: Slot [7] registered [ 0.365999] acpiphp: Slot [8] registered [ 0.366949] acpiphp: Slot [9] registered [ 0.368070] acpiphp: Slot [10] registered [ 0.369114] acpiphp: Slot [11] registered [ 0.370074] acpiphp: Slot [12] registered [ 0.372070] acpiphp: Slot [13] registered [ 0.373068] acpiphp: Slot [14] registered [ 0.374066] acpiphp: Slot [15] registered [ 0.376103] acpiphp: Slot [16] registered [ 0.377113] acpiphp: Slot [17] registered [ 0.378102] acpiphp: Slot [18] registered [ 0.380079] acpiphp: Slot [19] registered [ 0.382079] acpiphp: Slot [20] registered [ 0.383080] acpiphp: Slot [21] registered [ 0.385109] acpiphp: Slot [22] registered [ 0.386076] acpiphp: Slot [23] registered [ 0.388089] acpiphp: Slot [24] registered [ 0.389094] acpiphp: Slot [25] registered [ 0.391112] acpiphp: Slot [26] registered [ 0.392075] acpiphp: Slot [27] registered [ 0.394074] acpiphp: Slot [28] registered [ 0.395071] acpiphp: Slot [29] registered [ 0.397103] acpiphp: Slot [30] registered [ 0.398076] acpiphp: Slot [31] registered [ 0.399056] PCI host bridge to bus 0000:00 [ 0.401017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.402007] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.403009] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.405012] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.406009] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.408013] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.409187] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.410924] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.412804] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.418511] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.422000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.422010] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.423000] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.425013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.426681] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.428542] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.430066] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.432622] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.437009] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.449016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.452008] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.456784] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.462023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.466013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.475019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.482959] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.487010] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.492013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.507013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.515239] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.517414] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.520551] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.522327] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.525239] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.532099] iommu: Default domain type: Passthrough [ 0.533518] SCSI subsystem initialized [ 0.535137] ACPI: bus type USB registered [ 0.538143] usbcore: registered new interface driver usbfs [ 0.541100] usbcore: registered new interface driver hub [ 0.543071] usbcore: registered new device driver usb [ 0.545188] pps_core: LinuxPPS API ver. 1 registered [ 0.547011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.550076] PTP clock support registered [ 0.552147] EDAC MC: Ver: 3.0.0 [ 0.553274] PCI: Using ACPI for IRQ routing [ 0.554618] NetLabel: Initializing [ 0.555005] NetLabel: domain hash size = 128 [ 0.556007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.557073] NetLabel: unlabeled traffic allowed by default [ 0.558091] vgaarb: loaded [ 0.559216] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.560007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.566727] clocksource: Switched to clocksource kvm-clock [ 0.663864] VFS: Disk quotas dquot_6.6.0 [ 0.664899] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.666523] *** VALIDATE ramfs *** [ 0.667757] *** VALIDATE hugetlbfs *** [ 0.669399] pnp: PnP ACPI init [ 0.672393] pnp: PnP ACPI: found 6 devices [ 0.684602] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.686525] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.688102] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.689960] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.691560] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.692861] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.694543] NET: Registered protocol family 2 [ 0.696541] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.700429] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.702807] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.706429] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.708730] TCP: Hash tables configured (established 65536 bind 65536) [ 0.710731] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.712829] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.715058] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.717532] NET: Registered protocol family 1 [ 0.720333] RPC: Registered named UNIX socket transport module. [ 0.722770] RPC: Registered udp transport module. [ 0.724399] RPC: Registered tcp transport module. [ 0.726121] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.728493] NET: Registered protocol family 44 [ 0.730136] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.732426] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.734674] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.737056] PCI: CLS 0 bytes, default 64 [ 0.738470] Unpacking initramfs... [ 2.592433] debug: unmapping init [mem 0xffff8c4e7cc64000-0xffff8c4e7ffcffff] [ 2.596676] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.599190] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.602281] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.422620] Initialise system trusted keyrings [ 3.424242] Key type blacklist registered [ 3.425921] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.436725] zbud: loaded [ 3.439843] *** VALIDATE nfs *** [ 3.440814] *** VALIDATE nfs4 *** [ 3.442220] pstore: using deflate compression [ 3.446296] Platform Keyring initialized [ 3.568504] NET: Registered protocol family 38 [ 3.570206] Key type asymmetric registered [ 3.572555] Asymmetric key parser 'x509' registered [ 3.574847] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.578607] io scheduler mq-deadline registered [ 3.580249] io scheduler kyber registered [ 3.582377] io scheduler bfq registered [ 3.584974] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.589338] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.593171] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.597152] ACPI: Power Button [PWRF] [ 3.605866] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.614661] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.634563] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.666169] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.695939] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.700918] Non-volatile memory driver v1.3 [ 3.703107] Linux agpgart interface v0.103 [ 3.738536] virtio_blk virtio1: [vda] 134792 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.742552] vda: detected capacity change from 0 to 69013504 [ 3.782037] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.785023] vdb: detected capacity change from 0 to 1073741824 [ 3.800835] libphy: Fixed MDIO Bus: probed [ 3.809357] usbcore: registered new interface driver usbserial_generic [ 3.811234] usbserial: USB Serial support registered for generic [ 3.812671] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.816153] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.817412] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.819327] mousedev: PS/2 mouse device common for all mice [ 3.822341] rtc_cmos 00:05: RTC can wake from S4 [ 3.824658] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.826528] rtc_cmos 00:05: registered as rtc0 [ 3.830343] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.832726] intel_pstate: CPU model not supported [ 3.833928] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.839526] hid: raw HID events driver (C) Jiri Kosina [ 3.841669] usbcore: registered new interface driver usbhid [ 3.843072] usbhid: USB HID core driver [ 3.844929] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.846256] drop_monitor: Initializing network drop monitor service [ 3.851039] Initializing XFRM netlink socket [ 3.852940] NET: Registered protocol family 10 [ 3.855119] Segment Routing with IPv6 [ 3.855947] NET: Registered protocol family 17 [ 3.857204] mpls_gso: MPLS GSO support [ 3.860752] RAS: Correctable Errors collector initialized. [ 3.862133] AVX version of gcm_enc/dec engaged. [ 3.863156] AES CTR mode by8 optimization enabled [ 4.011602] sched_clock: Marking stable (4011525937, 0)->(4721621774, -710095837) [ 4.015857] registered taskstats version 1 [ 4.020559] Loading compiled-in X.509 certificates [ 4.024888] zswap: loaded using pool lzo/zbud [ 4.052330] Key type big_key registered [ 4.068480] Key type encrypted registered [ 4.070285] ima: No TPM chip found, activating TPM-bypass! [ 4.072937] ima: Allocated hash algorithm: sha1 [ 4.075258] ima: No architecture policies found [ 4.077764] evm: Initialising EVM extended attributes: [ 4.080688] evm: security.selinux [ 4.082590] evm: security.ima [ 4.083929] evm: security.capability [ 4.086259] evm: HMAC attrs: 0x1 [ 4.089992] rtc_cmos 00:05: setting system clock to 2026-05-20 20:21:34 UTC (1779308494) [ 4.101650] debug: unmapping init [mem 0xffffffff8ec03000-0xffffffff8edfffff] [ 4.107765] debug: unmapping init [mem 0xffffffff8d982000-0xffffffff8dc58fff] [ 4.118541] Write protecting the kernel read-only data: 28672k [ 4.125236] debug: unmapping init [mem 0xffffffff8c003000-0xffffffff8c1fffff] [ 4.129480] debug: unmapping init [mem 0xffffffff8c914000-0xffffffff8c9fffff] [ 4.199797] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.209395] systemd[1]: Detected virtualization kvm. [ 4.211744] systemd[1]: Detected architecture x86-64. [ 4.215511] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.252728] systemd[1]: No hostname configured. [ 4.254520] systemd[1]: Set hostname to . [ 4.256893] random: systemd: uninitialized urandom read (16 bytes read) [ 4.259753] systemd[1]: Initializing machine ID from random generator. [ 4.385246] random: ln: uninitialized urandom read (6 bytes read) [ 4.795137] random: systemd: uninitialized urandom read (16 bytes read) [ 4.799203] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.810803] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.819617] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.905317] device-mapper: uevent: version 1.0.3 [ 5.907534] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 8.276700] virtio_net virtio0 ens2: renamed from eth0 [ 8.630920] random: fast init done [ 8.697282] scsi host0: ata_piix [ 8.920712] scsi host1: ata_piix [ 8.922583] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 8.926223] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 13.631875] random: crng init done [ 13.634307] random: 7 urandom warning(s) missed due to ratelimiting [ 16.153887] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.929351] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.403480] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.769058] SELinux: Disabled at runtime. [ 18.866178] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.875721] systemd[1]: Detected virtualization kvm. [ 18.877652] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.948925] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.955259] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.995877] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.003843] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.009466] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.018521] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.031991] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ 20.296882] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 20.977876] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.719103] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.857386] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.228091] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.314264] EDAC sbridge: Ver: 1.1.2 [ 24.963899] Key type dns_resolver registered [ 25.329436] NFS: Registering the id_resolver key type [ 25.331165] Key type id_resolver registered [ 25.332371] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg253-client login: [ 69.635437] libcfs: loading out-of-tree module taints kernel. [ 69.656100] Key type ._llcrypt registered [ 69.657251] Key type .llcrypt registered [ 69.961611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 69.969441] alg: No test for adler32 (adler32-zlib) [ 70.978670] Lustre: Lustre: Build Version: 2.17.53_24_ge4eab26 [ 71.253020] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 72.871175] Key type lgssc registered [ 73.482666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 128.259976] Lustre: Mounted lustre-client [ 131.012546] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 137.114524] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing check_logdir /tmp/testlogs/ [ 139.201194] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing yml_node [ 141.286784] Lustre: DEBUG MARKER: Client: 2.17.53.24 [ 142.252344] Lustre: DEBUG MARKER: MDS: 2.17.53.24 [ 143.390225] Lustre: DEBUG MARKER: OSS: 2.17.53.24 [ 144.197372] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed May 20 16:23:53 EDT 2026 [ 150.761144] LustreError: 7938:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8c4ec7a1e000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 150.776883] Lustre: Unmounted lustre-client [ 177.858975] Key type lgssc unregistered [ 177.986509] LNet: 8384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 177.989916] LNetError: 8384:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 177.997537] LNet: Removed LNI 192.168.202.53@tcp [ 178.267168] Key type .llcrypt unregistered [ 178.268397] Key type ._llcrypt unregistered [ 187.260343] Key type ._llcrypt registered [ 187.261886] Key type .llcrypt registered [ 187.342858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 187.350024] alg: No test for adler32 (adler32-zlib) [ 188.182288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 188.213490] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 188.229115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 189.791859] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 189.859059] LNet: 9255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 189.866413] LNetError: 9255:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 189.881571] LNet: Removed LNI 192.168.202.53@tcp [ 190.204446] Key type .llcrypt unregistered [ 190.205623] Key type ._llcrypt unregistered [ 192.453743] Key type ._llcrypt registered [ 192.455315] Key type .llcrypt registered [ 193.037269] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 16:24:42 (1779308682) [ 193.165053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 193.170885] alg: No test for adler32 (adler32-zlib) [ 193.983329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 194.024662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 194.034422] LNet: 9670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 194.041971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 194.059619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 194.066408] LNet: 9677:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 196.588345] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 16:24:46 (1779308686) [ 196.621834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 196.633452] LNet: 10248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 196.645968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 196.694234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 199.361564] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 16:24:49 (1779308689) [ 199.431259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 199.443207] LNet: 10836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 199.457730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 199.500334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 202.215356] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 16:24:52 (1779308692) [ 202.246795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 202.257040] LNet: 11415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 202.269161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 202.305579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 205.327341] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 16:24:55 (1779308695) [ 205.409826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 205.438993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 205.496669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 205.695328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 208.465209] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 16:24:58 (1779308698) [ 208.518612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 208.529580] LNet: 12592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 208.532712] LNet: 12592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 208.544017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 208.581221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 208.760429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 208.935988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 211.749916] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 16:25:01 (1779308701) [ 211.806716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 211.825810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 211.866995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 214.488670] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 16:25:04 (1779308704) [ 214.527390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 214.551432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.587195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 214.605504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 214.761962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 214.787683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 214.946390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 214.964987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 215.144818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 215.165282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 215.363534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 215.385650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 215.555778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 215.574500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 215.730712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 215.745539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 215.887523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 215.902256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 216.071729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 216.086792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 216.249173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 216.272736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 216.448420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 216.465459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 216.643465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 216.669833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 216.819756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 216.836109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 219.719204] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 16:25:09 (1779308709) [ 219.756884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 219.775103] LNet: 14449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 219.782161] LNet: 14449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 219.797449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 219.838649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 219.857743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 222.248363] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 16:25:12 (1779308712) [ 222.309391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.331546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.374196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 222.395591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 224.941508] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 16:25:14 (1779308714) [ 225.022433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 225.036864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 225.064835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 225.078850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 227.532295] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 16:25:17 (1779308717) [ 227.599396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 227.621367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 227.656305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 227.669953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 230.357890] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 16:25:20 (1779308720) [ 230.398399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 230.415175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.455648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 230.469773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 233.204175] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 16:25:23 (1779308723) [ 233.240908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.261349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.299022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 233.318419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 236.027616] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 16:25:25 (1779308725) [ 236.073615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 236.092299] LNet: 17961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 236.094489] LNet: 17961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 236.102807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 236.149920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 236.322826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 238.814942] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 16:25:28 (1779308728) [ 238.877357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.900488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.973222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 239.070171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 241.629774] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 16:25:31 (1779308731) [ 241.717725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 241.733776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 241.790773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 241.872743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 244.269347] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 16:25:34 (1779308734) [ 244.297111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.313755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.488777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 244.586750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 247.113197] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 16:25:37 (1779308737) [ 247.184087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.202693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.233672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 247.255758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 249.647392] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 16:25:39 (1779308739) [ 249.676984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.692314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.727927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 249.748659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 252.129366] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 16:25:41 (1779308741) [ 252.158650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 252.176410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 252.211085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 254.757474] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 16:25:44 (1779308744) [ 254.788102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 254.807337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 254.849562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 257.365294] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 16:25:47 (1779308747) [ 257.422937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 257.442032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 257.476369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 257.677803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 260.117119] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 16:25:49 (1779308749) [ 260.152766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 260.169980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.201685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 260.215625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 262.541729] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 16:25:52 (1779308752) [ 262.596122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 262.614662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 262.646335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 262.662310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 264.966315] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 16:25:54 (1779308754) [ 265.014684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 265.033300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.061762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 265.075547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 267.139339] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 16:25:57 (1779308757) [ 267.175288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 267.191455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 267.219572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 267.234773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 269.350790] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 16:25:59 (1779308759) [ 269.380663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 269.389099] LNet: 25738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 269.392069] LNet: 25738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 269.402480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 269.427434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 269.450184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 269.457940] LNetError: 25756:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 269.476866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 269.491121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 271.393479] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 16:26:01 (1779308761) [ 271.421321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.438313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.472131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.PAyHy2wh3I [ 271.494553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.PAyHy2wh3I [ 271.516653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.PAyHy2wh3I [ 273.346553] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 16:26:03 (1779308763) [ 273.384630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.401335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.418380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 273.429486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 273.444290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 275.314454] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 16:26:05 (1779308765) [ 275.347944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 275.361057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 275.384164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 276.302881] LNet: Message forwarding will be enabled in 2s [ 279.391133] LNet: Message forwarding enabled [ 279.423144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 279.431927] LNet: Message forwarding will stop in 2s to 6s [ 283.565127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 284.303792] LNet: Message forwarding disabled [ 284.746310] LNet: Message forwarding will be enabled in 2s [ 286.721356] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 16:26:16 (1779308776) [ 286.758800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 287.346459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 287.368630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 289.298535] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 16:26:19 (1779308779) [ 289.337630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.357455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.384436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 289.396063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 289.408268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 289.421916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 289.436882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 289.451138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 289.464670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 289.479626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 289.495880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 289.511305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 289.523886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 289.536762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 289.550940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 289.565604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 289.577401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 289.591297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 289.608796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 289.623401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 289.639138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 289.648512] LNetError: 28821:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 291.704122] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 16:26:21 (1779308781) [ 291.756076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 291.774956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 291.807959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 294.011160] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 16:26:23 (1779308783) [ 294.642249] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg253-server [ 295.265250] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 16:26:25 (1779308785) [ 295.295209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.315799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.346285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 295.356252] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 295.950343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[54-58]@tcp [ 295.960391] LNet: 30535:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 295.970207] LNetError: 30324:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.55@tcp is being used as a gateway but routing feature is not turned on [ 295.975179] LNetError: 30324:0:(lib-lnet.h:1268:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.55@tcp has gone from up to down [ 296.993284] LNetError: 30324:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.55@tcp is being used as a gateway but routing feature is not turned on [ 296.999134] LNetError: 30324:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 298.017108] LNetError: 30324:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.55@tcp is being used as a gateway but routing feature is not turned on [ 298.224214] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 16:26:28 (1779308788) [ 298.833857] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg253-server [ 299.040413] LNetError: 30330:0:(lib-lnet.h:1268:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.58@tcp has gone from up to down [ 299.379970] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 16:26:29 (1779308789) [ 299.416323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.427165] LNetError: 31466:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 299.436716] LNet: Removed LNI 192.168.202.53@tcp [ 299.445370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.465614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 299.476245] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 299.698499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[54-59/2]@tcp [ 299.708461] LNet: 31586:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 299.713756] LNet: 31586:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 299.722909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[54-59/2]@tcp [ 301.731692] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 16:26:31 (1779308791) [ 301.768676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 302.815814] LNetError: 32172:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 302.824855] LNet: Removed LNI 192.168.202.53@tcp [ 302.835504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.866361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 302.888800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 302.901307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.923721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 302.946884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 302.960713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.978494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 303.003958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.016827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.039566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 303.067638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.079282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.096268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 303.123322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.135317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.161023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 303.191902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.205761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.222547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 303.248687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.262083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.283305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 303.309814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.327801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.349435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 303.372562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.383342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.401966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 303.426188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.438961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.457504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 303.479754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.493062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.515399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 305.566206] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 16:26:35 (1779308795) [ 306.158792] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg253-server [ 306.721352] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 16:26:36 (1779308796) [ 307.292839] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg253-server [ 307.862942] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 16:26:37 (1779308797) [ 308.175246] Key type .llcrypt unregistered [ 308.176501] Key type ._llcrypt unregistered [ 310.170759] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 310.250358] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 310.252417] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 310.254696] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 312.405863] Key type ._llcrypt registered [ 312.406825] Key type .llcrypt registered [ 312.460938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 312.466218] alg: No test for adler32 (adler32-zlib) [ 313.265933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.286794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 313.295037] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 313.501794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 313.507990] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 314.111774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 314.118267] LNet: Removed LNI 10.1.2.3@tcp [ 314.155129] LNet: 34296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 314.158629] LNetError: 34296:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 314.166314] LNet: Removed LNI 192.168.202.53@tcp [ 314.366299] Key type .llcrypt unregistered [ 314.367437] Key type ._llcrypt unregistered [ 317.727974] Key type ._llcrypt registered [ 317.728883] Key type .llcrypt registered [ 318.154494] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 16:26:48 (1779308808) [ 318.318305] Key type .llcrypt unregistered [ 318.319467] Key type ._llcrypt unregistered [ 320.375595] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 320.460697] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 320.462698] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 320.465415] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 322.785413] Key type ._llcrypt registered [ 322.789067] Key type .llcrypt registered [ 322.910319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 322.923813] alg: No test for adler32 (adler32-zlib) [ 323.779935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.826951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 323.840772] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 324.183756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 324.194725] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 325.318654] LNet: 35800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.322320] LNetError: 35800:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 325.325602] LNet: Removed LNI 192.168.202.53@tcp [ 325.333583] LNet: Removed LNI 10.1.2.3@tcp [ 325.604473] Key type .llcrypt unregistered [ 325.605812] Key type ._llcrypt unregistered [ 332.992234] Key type ._llcrypt registered [ 332.993844] Key type .llcrypt registered [ 333.871968] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 16:27:03 (1779308823) [ 334.384321] Key type .llcrypt unregistered [ 334.386245] Key type ._llcrypt unregistered [ 339.198121] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 339.348729] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 339.351586] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 339.360185] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 341.758855] Key type ._llcrypt registered [ 341.760799] Key type .llcrypt registered [ 341.917452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 341.929709] alg: No test for adler32 (adler32-zlib) [ 342.804156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.838758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 342.855379] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 343.997390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 344.010255] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 345.400246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 345.417529] LNet: Removed LNI 10.1.2.3@tcp [ 345.428653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 345.474546] LNet: Removed LNI 10.1.2.31@tcp [ 345.486428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 345.509288] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 346.024762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 346.050209] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 347.068399] LNet: 37714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 347.077903] LNetError: 37714:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 347.087284] LNet: Removed LNI 10.1.2.3@tcp [ 347.098496] LNet: Removed LNI 10.1.2.31@tcp [ 347.589445] Key type .llcrypt unregistered [ 347.591321] Key type ._llcrypt unregistered [ 358.958113] Key type ._llcrypt registered [ 358.959808] Key type .llcrypt registered [ 360.257635] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 16:27:29 (1779308849) [ 360.508856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 360.517717] alg: No test for adler32 (adler32-zlib) [ 361.376353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.451643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 361.484763] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 362.558456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.575899] LNet: 38668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.588869] LNetError: 38668:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 362.611774] LNet: Removed LNI 192.168.202.53@tcp [ 362.623843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.655920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.53@tcp [ 362.669438] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 363.220931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.53@tcp [ 363.248813] LNet: Removed LNI 192.168.202.53@tcp [ 363.293225] LNet: 38803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.685508] Key type .llcrypt unregistered [ 363.687942] Key type ._llcrypt unregistered [ 370.159578] Key type ._llcrypt registered [ 370.160953] Key type .llcrypt registered [ 370.797798] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 16:27:40 (1779308860) [ 371.895808] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg253-server [ 372.779872] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 16:27:42 (1779308862) [ 373.166340] Key type .llcrypt unregistered [ 373.168713] Key type ._llcrypt unregistered [ 376.862062] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 376.891336] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 376.894155] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 376.901405] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 379.275268] Key type ._llcrypt registered [ 379.276970] Key type .llcrypt registered [ 379.392923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 379.401973] alg: No test for adler32 (adler32-zlib) [ 380.239590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.278049] LNet: Added LNI 192.168.202.53@tcp1 [8/256/0/180] [ 380.653812] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 381.926624] LNet: 40636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.930937] LNetError: 40636:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 381.935956] LNet: Removed LNI 192.168.202.53@tcp1 [ 381.944241] LNet: Removed LNI 10.1.2.3@tcp2 [ 382.318505] Key type .llcrypt unregistered [ 382.320464] Key type ._llcrypt unregistered [ 389.292446] Key type ._llcrypt registered [ 389.293627] Key type .llcrypt registered [ 390.054543] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 16:27:59 (1779308879) [ 390.232747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 390.240694] alg: No test for adler32 (adler32-zlib) [ 391.069391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.102880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 391.117579] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 391.588857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.600981] LNet: 41603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.607158] LNetError: 41603:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 391.616601] LNet: Removed LNI 192.168.202.53@tcp [ 391.631865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.665693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 391.678355] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 395.496559] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 16:28:05 (1779308885) [ 395.556325] LNet: 42185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.560517] LNetError: 42185:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 395.576312] LNet: Removed LNI 192.168.202.53@tcp [ 395.943750] Key type .llcrypt unregistered [ 395.944963] Key type ._llcrypt unregistered [ 398.484732] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 398.606848] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 398.609550] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 398.615484] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 400.977732] Key type ._llcrypt registered [ 400.978883] Key type .llcrypt registered [ 401.086975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 401.098924] alg: No test for adler32 (adler32-zlib) [ 401.919789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.951346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 401.965592] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 402.337926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 402.348609] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 403.499575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.509326] LNet: 42937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.521319] LNetError: 42937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 403.524553] LNet: Removed LNI 192.168.202.53@tcp [ 403.538632] LNet: Removed LNI 10.1.2.3@tcp [ 403.548313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.573593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 403.587567] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 403.592216] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 403.785459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.793070] LNet: Removed LNI 192.168.202.53@tcp [ 403.801913] LNet: Removed LNI 10.1.2.3@tcp [ 403.818386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.853143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 403.874413] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 403.881707] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 404.125639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 404.143980] LNet: Added LNI 192.168.202.53@tcp2 [8/256/0/180] [ 404.680949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.692466] LNet: 43126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.695296] LNet: 43126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 404.700274] LNetError: 43126:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 404.703503] LNetError: 43126:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 404.710525] LNet: Removed LNI 192.168.202.53@tcp [ 404.712371] LNet: Removed LNI 10.1.2.3@tcp [ 404.726708] LNet: Removed LNI 192.168.202.53@tcp2 [ 404.739277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.772542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 404.789719] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 404.795727] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 404.800254] LNet: Added LNI 192.168.202.53@tcp2 [8/256/0/180] [ 408.194883] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 16:28:17 (1779308897) [ 408.266440] LNet: 43718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.272061] LNetError: 43718:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 408.275842] LNetError: 43718:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 408.282494] LNet: Removed LNI 192.168.202.53@tcp [ 408.284139] LNet: Removed LNI 10.1.2.3@tcp [ 408.290845] LNet: Removed LNI 192.168.202.53@tcp2 [ 408.670785] Key type .llcrypt unregistered [ 408.674084] Key type ._llcrypt unregistered [ 412.161537] Key type ._llcrypt registered [ 412.162891] Key type .llcrypt registered [ 412.269198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 412.281690] alg: No test for adler32 (adler32-zlib) [ 413.132752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.165830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 413.181251] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 413.599571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.614179] LNet: 44117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.618853] LNetError: 44117:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 413.635413] LNet: Removed LNI 192.168.202.53@tcp [ 413.652979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.696933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 413.712251] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 414.007741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.028416] LNet: Removed LNI 192.168.202.53@tcp [ 414.041869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.081736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 414.104393] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 414.389405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.408726] LNet: 44177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.413193] LNet: 44177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 414.418584] LNetError: 44177:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 414.421566] LNetError: 44177:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 414.444821] LNet: Removed LNI 192.168.202.53@tcp [ 414.465692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.506165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 414.520646] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 414.766632] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 414.902407] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 414.909282] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 414.912334] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 417.070797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.082506] LNet: 44250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.085914] LNetError: 44250:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 417.093726] LNet: Removed LNI 192.168.202.53@tcp [ 417.104889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.152707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 417.167741] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 420.459109] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 16:28:30 (1779308910) [ 420.528769] LNet: 44839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.538363] LNetError: 44839:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 420.560298] LNet: Removed LNI 192.168.202.53@tcp [ 420.988091] Key type .llcrypt unregistered [ 420.990507] Key type ._llcrypt unregistered [ 423.642269] Key type ._llcrypt registered [ 423.643580] Key type .llcrypt registered [ 423.731541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 423.738995] alg: No test for adler32 (adler32-zlib) [ 424.562628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.591415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 424.608544] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 425.059940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.070440] LNet: 45244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.073106] LNetError: 45244:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 425.080984] LNet: Removed LNI 192.168.202.53@tcp [ 425.094256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.120872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 425.132819] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 428.525601] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 16:28:38 (1779308918) [ 428.585120] LNet: 45830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 428.589767] LNetError: 45830:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 428.603234] LNet: Removed LNI 192.168.202.53@tcp [ 428.977596] Key type .llcrypt unregistered [ 428.978957] Key type ._llcrypt unregistered [ 431.436577] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 431.585972] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 431.589268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 431.592811] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 433.889835] Key type ._llcrypt registered [ 433.891945] Key type .llcrypt registered [ 434.020711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 434.037475] alg: No test for adler32 (adler32-zlib) [ 434.862593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.896304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 434.916864] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 435.331564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.345576] LNet: 46282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 435.348688] LNetError: 46282:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 435.365922] LNet: Removed LNI 192.168.202.53@tcp [ 435.378910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.410268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 435.433811] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 438.512341] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 16:28:48 (1779308928) [ 438.614403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.623928] LNet: 46879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 438.626984] LNetError: 46879:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 438.640777] LNet: Removed LNI 192.168.202.53@tcp [ 438.649557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.696084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 441.575227] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 16:28:51 (1779308931) [ 441.644196] LNet: 47456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 441.902541] Key type .llcrypt unregistered [ 441.903989] Key type ._llcrypt unregistered [ 444.364746] Key type ._llcrypt registered [ 444.365956] Key type .llcrypt registered [ 444.456474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 444.463144] alg: No test for adler32 (adler32-zlib) [ 445.281875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.306227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 445.316322] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 445.672607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.681498] LNet: 47858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 445.684206] LNetError: 47858:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 445.694352] LNet: Removed LNI 192.168.202.53@tcp [ 445.701784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.724383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 445.735424] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 448.868770] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 16:28:58 (1779308938) [ 448.914670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.931648] LNet: 48444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 448.935064] LNetError: 48444:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 448.941839] LNet: Removed LNI 192.168.202.53@tcp [ 448.951504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.987942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 451.599448] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 16:29:01 (1779308941) [ 452.291032] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 452.998145] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 16:29:02 (1779308942) [ 453.735964] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 454.552594] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 16:29:04 (1779308944) [ 455.223511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 455.954791] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 16:29:05 (1779308945) [ 456.591683] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 457.285458] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 16:29:07 (1779308947) [ 457.948750] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 458.696214] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 16:29:08 (1779308948) [ 458.740803] LNet: 50490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 459.046543] Key type .llcrypt unregistered [ 459.048275] Key type ._llcrypt unregistered [ 461.173671] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 461.237226] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 461.239601] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 461.241706] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 463.425095] Key type ._llcrypt registered [ 463.426617] Key type .llcrypt registered [ 463.501264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 463.507661] alg: No test for adler32 (adler32-zlib) [ 464.329443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.375530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 464.392864] LNet: Added LNI 192.168.202.53@tcp [8/512/0/180] [ 464.395773] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 464.397017] LNet: 50813:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 464.439169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.450123] LNet: 50832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.453798] LNetError: 50832:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 464.456903] LNet: Removed LNI 192.168.202.53@tcp [ 464.458461] LNetError: 50832:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 464.468447] LNet: Removed LNI 10.1.2.3@tcp [ 464.478452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.517032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 464.527414] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 464.562426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.572351] LNet: 50856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.575385] LNetError: 50856:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 464.584417] LNet: Removed LNI 192.168.202.53@tcp [ 464.596797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.647610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 464.660966] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 464.664503] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 464.667426] LNet: 50864:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 464.707860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.717607] LNet: 50882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.720219] LNetError: 50882:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 464.722025] LNet: Removed LNI 192.168.202.53@tcp [ 464.722975] LNetError: 50882:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 464.735403] LNet: Removed LNI 10.1.2.3@tcp [ 464.745703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.784224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 464.796995] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 464.799843] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 464.801820] LNet: 50891:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 464.896919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.908807] LNet: 50917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.912632] LNetError: 50917:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 464.916070] LNet: Removed LNI 192.168.202.53@tcp [ 464.918195] LNetError: 50917:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 464.925571] LNet: Removed LNI 10.1.2.3@tcp [ 464.938121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.972696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 464.991578] LNet: Added LNI 192.168.202.53@tcp [8/256/0/182] [ 464.995214] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 465.050846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.061479] LNet: 50945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 465.064613] LNetError: 50945:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 465.068384] LNet: Removed LNI 192.168.202.53@tcp [ 465.070466] LNetError: 50945:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 465.079553] LNet: Removed LNI 10.1.2.3@tcp [ 465.090637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.128280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 465.140622] LNet: Added LNI 192.168.202.53@tcp [8/256/0/182] [ 465.143164] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 465.145243] LNet: 50953:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 465.186917] LNet: Removed LNI 192.168.202.53@tcp [ 465.196647] LNet: Removed LNI 10.1.2.3@tcp [ 465.473042] Key type .llcrypt unregistered [ 465.474863] Key type ._llcrypt unregistered [ 469.742657] Key type ._llcrypt registered [ 469.743801] Key type .llcrypt registered [ 470.426435] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 16:29:20 (1779308960) [ 470.613599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 470.621168] alg: No test for adler32 (adler32-zlib) [ 471.459279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.487213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 471.498929] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 471.857682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.53 [ 471.868573] LNet: 51927:0:(config.c:1628:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 471.880819] LNet: Removed LNI 192.168.202.53@tcp [ 474.852930] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 16:29:24 (1779308964) [ 474.897340] LNet: 52488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.902206] LNetError: 52488:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 474.917538] LNet: Removed LNI 192.168.202.53@tcp [ 475.191286] Key type .llcrypt unregistered [ 475.193280] Key type ._llcrypt unregistered [ 477.572868] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 477.672158] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 479.957383] Key type ._llcrypt registered [ 479.959057] Key type .llcrypt registered [ 480.024603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 480.031925] alg: No test for adler32 (adler32-zlib) [ 481.049143] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 483.849369] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 16:29:33 (1779308973) [ 483.952426] LNet: 53408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 483.955991] LNetError: 53408:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 483.968244] LNet: Removed LNI 10.1.2.3@tcp [ 484.225450] Key type .llcrypt unregistered [ 484.227543] Key type ._llcrypt unregistered [ 486.598036] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 486.721054] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 486.724548] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 486.729054] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 489.138647] Key type ._llcrypt registered [ 489.140748] Key type .llcrypt registered [ 489.228445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 489.241596] alg: No test for adler32 (adler32-zlib) [ 490.071181] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 493.328826] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 16:29:43 (1779308983) [ 494.024169] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 494.881440] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 16:29:44 (1779308984) [ 495.005357] LNet: 54591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 495.010109] LNetError: 54591:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 495.019130] LNet: Removed LNI 10.1.2.3@tcp [ 495.387353] Key type .llcrypt unregistered [ 495.388983] Key type ._llcrypt unregistered [ 498.135820] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 498.305741] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 498.308832] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 498.310937] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 500.718103] Key type ._llcrypt registered [ 500.719942] Key type .llcrypt registered [ 500.841572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 500.848286] alg: No test for adler32 (adler32-zlib) [ 501.691896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.737120] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 506.180359] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 16:29:55 (1779308995) [ 506.399987] LNet: 55818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 506.403928] LNetError: 55818:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 506.415989] LNet: Removed LNI 10.1.2.3@tcp [ 506.704666] Key type .llcrypt unregistered [ 506.706557] Key type ._llcrypt unregistered [ 509.808105] Key type ._llcrypt registered [ 509.809538] Key type .llcrypt registered [ 509.937358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 509.962718] alg: No test for adler32 (adler32-zlib) [ 510.799729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 510.817705] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 514.408894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.153@tcp [ 514.749958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_interrupt -r 1 [ 514.771830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_interrupt -r 1 [ 514.798672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 514.813600] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 514.912120] LNet: 1 local NIs in recovery (showing 1): 192.168.202.53@tcp [ 514.941732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 514.966100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 514.983961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 515.006104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 515.030635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 515.049379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 515.067935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 515.085981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 515.096061] LNet: 56255:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 515.106711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 515.126464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 515.144568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 515.161627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 515.181247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 515.205343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 515.231202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 515.539708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_dropped -r 1 [ 515.572102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_dropped -r 1 [ 515.606960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 515.623418] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 515.775785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 515.792301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 515.807444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 515.830902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 515.845734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 515.858846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 515.873444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 515.887852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 515.896465] LNet: 56364:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 515.907472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 515.928982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 515.953882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 515.970917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 515.990666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 516.013410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 516.043707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 516.359833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_aborted -r 1 [ 516.383624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_aborted -r 1 [ 516.401605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 516.540273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 516.559181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 516.576142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 516.592248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 516.608975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 516.627717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 516.644727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 516.660949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 516.675614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 516.692187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 516.707304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 516.734484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 516.749963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 516.765712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 516.784279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 517.048804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_no_route -r 1 [ 517.067973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_no_route -r 1 [ 517.089500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 517.100882] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 517.103034] LNet: Skipped 2 previous similar messages [ 517.203912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 517.221412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 517.253942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 517.273717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 517.295207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 517.314902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 517.336614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 517.359611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 517.375501] LNet: 56582:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 517.380237] LNet: 56582:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 517.391521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 517.411219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 517.433755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 517.447975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 517.464869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 517.479418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 517.511560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 517.804378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_timeout -r 1 [ 517.825869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_timeout -r 1 [ 517.849342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 518.003901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 518.020537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 518.040562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 518.053376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 518.067645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 518.082061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 518.095748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 518.110693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 518.126250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 518.141086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 518.154620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 518.180974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 518.206936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 518.230422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 518.256321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 518.554989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_error -r 1 [ 518.571880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_error -r 1 [ 518.600512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 518.734747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 518.756506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 518.773961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 518.793370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 518.808699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 518.827573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 518.845155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 518.871742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 518.887258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 518.900360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 518.921770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 518.939439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 518.954771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 518.980256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 518.997626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 519.147762] LNet: 56842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 519.153901] LNetError: 56842:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 520.173233] LNet: Removed LNI 192.168.202.53@tcp [ 520.422691] Key type .llcrypt unregistered [ 520.425496] Key type ._llcrypt unregistered [ 527.990740] Key type ._llcrypt registered [ 527.991988] Key type .llcrypt registered [ 528.682295] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 16:30:18 (1779309018) [ 529.107385] Key type .llcrypt unregistered [ 529.108971] Key type ._llcrypt unregistered [ 531.421827] Key type ._llcrypt registered [ 531.423777] Key type .llcrypt registered [ 531.526854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 531.534526] alg: No test for adler32 (adler32-zlib) [ 532.361525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 532.377401] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 535.951632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.153@tcp [ 539.938757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.957315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 539.970830] LNet: Added LNI 192.168.202.53@tcp1 [8/256/0/180] [ 540.699777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_interrupt -r 1 [ 540.723588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp1 -m GET -e local_interrupt -r 1 [ 540.749563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_interrupt -r 1 [ 540.770586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -m GET -e local_interrupt -r 1 [ 540.801712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp -m GET -e local_interrupt -r 1 [ 540.829773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp1 -m GET -e local_interrupt -r 1 [ 540.851488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -m GET -e local_interrupt -r 1 [ 540.868582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -m GET -e local_interrupt -r 1 [ 540.887656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 540.901198] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 540.903787] LNet: 1 local NIs in recovery (showing 1): 192.168.202.53@tcp [ 541.015682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 541.031439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 541.045671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 541.059221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 541.077917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 541.103270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 541.122277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 541.142319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 541.151982] LNet: 58329:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 541.163550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 541.206353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 541.231343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 541.252407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 541.276250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 541.301508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 541.335485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 541.675553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_dropped -r 1 [ 541.713740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp1 -m GET -e local_dropped -r 1 [ 541.742777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_dropped -r 1 [ 541.764889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -m GET -e local_dropped -r 1 [ 541.785183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp -m GET -e local_dropped -r 1 [ 541.802912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp1 -m GET -e local_dropped -r 1 [ 541.820663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -m GET -e local_dropped -r 1 [ 541.845918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -m GET -e local_dropped -r 1 [ 541.867575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 541.880395] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 541.883734] LNet: Skipped 2 previous similar messages [ 541.990859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 542.006831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 542.025550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 542.045475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 542.063698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 542.079498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 542.100204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 542.121817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 542.130970] LNet: 58450:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 542.141756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 542.160935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 542.181663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 542.199362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 542.217674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 542.231624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 542.255284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 542.504578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_aborted -r 1 [ 542.524884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp1 -m GET -e local_aborted -r 1 [ 542.544624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_aborted -r 1 [ 542.563763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -m GET -e local_aborted -r 1 [ 542.588770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp -m GET -e local_aborted -r 1 [ 542.624647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp1 -m GET -e local_aborted -r 1 [ 542.659125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -m GET -e local_aborted -r 1 [ 542.695287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -m GET -e local_aborted -r 1 [ 542.718706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 542.885289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 542.912468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 542.933365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 542.951360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 542.971212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 542.990842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 543.010672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 543.029248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 543.046863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 543.065706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 543.085864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 543.108694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 543.130144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 543.149748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 543.171854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 543.428615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_no_route -r 1 [ 543.458593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp1 -m GET -e local_no_route -r 1 [ 543.481187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_no_route -r 1 [ 543.510314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -m GET -e local_no_route -r 1 [ 543.534762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp -m GET -e local_no_route -r 1 [ 543.557522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp1 -m GET -e local_no_route -r 1 [ 543.581887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -m GET -e local_no_route -r 1 [ 543.604613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -m GET -e local_no_route -r 1 [ 543.628777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 543.641871] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 543.645782] LNet: Skipped 7 previous similar messages [ 543.770277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 543.790656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 543.815471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 543.844499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 543.864195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 543.890148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 543.915970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 543.941153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 543.954587] LNet: 58692:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 543.962505] LNet: 58692:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 543.975504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 543.990971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 544.007333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 544.024240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 544.040571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 544.062517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 544.084161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 544.386695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_timeout -r 1 [ 544.417608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp1 -m GET -e local_timeout -r 1 [ 544.439794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_timeout -r 1 [ 544.467069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -m GET -e local_timeout -r 1 [ 544.493830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp -m GET -e local_timeout -r 1 [ 544.518336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp1 -m GET -e local_timeout -r 1 [ 544.543710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -m GET -e local_timeout -r 1 [ 544.571887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -m GET -e local_timeout -r 1 [ 544.603233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 544.748809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 544.768616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 544.793804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 544.809924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 544.849255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 544.885932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 544.912809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 544.936506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 544.960316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 544.980501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 544.996664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 545.015660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 545.040245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 545.061613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 545.081238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 545.406172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_error -r 1 [ 545.439715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp1 -m GET -e local_error -r 1 [ 545.467664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_error -r 1 [ 545.501477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -m GET -e local_error -r 1 [ 545.522915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp -m GET -e local_error -r 1 [ 545.546423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp1 -m GET -e local_error -r 1 [ 545.574229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -m GET -e local_error -r 1 [ 545.595790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -m GET -e local_error -r 1 [ 545.612725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 545.713873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 545.732230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 545.754605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 545.777195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 545.799486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 545.820670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 545.838660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 545.860350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 545.883371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 545.905297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 545.924085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 545.947831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 545.969477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 545.994221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 546.016400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 548.558359] LNet: 59010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 548.564393] LNetError: 59010:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 548.567053] LNet: Removed LNI 192.168.202.53@tcp [ 549.605485] LNet: Removed LNI 192.168.202.53@tcp1 [ 549.818720] Key type .llcrypt unregistered [ 549.820059] Key type ._llcrypt unregistered [ 557.259318] Key type ._llcrypt registered [ 557.261113] Key type .llcrypt registered [ 557.985556] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 16:30:47 (1779309047) [ 558.367484] Key type .llcrypt unregistered [ 558.369874] Key type ._llcrypt unregistered [ 561.013567] Key type ._llcrypt registered [ 561.015142] Key type .llcrypt registered [ 561.137674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 561.148494] alg: No test for adler32 (adler32-zlib) [ 561.990539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 562.009406] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 565.573790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.153@tcp [ 565.920516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e remote_dropped -r 1 [ 565.953884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e remote_dropped -r 1 [ 565.977878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 565.991300] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 566.095943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 566.112681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 566.131093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 566.155066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 566.173118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 566.191301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 566.209735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 566.233983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 566.244210] LNet: 60256:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 566.256402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 566.273505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 566.294484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 566.325628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 566.350218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 566.379503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 566.401297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 566.775787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e remote_error -r 1 [ 566.797571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e remote_error -r 1 [ 566.820313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 566.835338] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 566.976047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 566.992860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 567.013241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 567.033120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 567.053718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 567.070796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 567.089497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 567.108467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 567.119947] LNet: 60365:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 567.139975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 567.166839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 567.185317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 567.205710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 567.222355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 567.243389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 567.266349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 567.632732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e remote_timeout -r 1 [ 567.648521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e remote_timeout -r 1 [ 567.668301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 567.798569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 567.829683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 567.853418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 567.877317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 567.900560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 567.914714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 567.931030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 567.949816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 567.970040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 567.989364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 568.009072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 568.027501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 568.047519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 568.080224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 568.103278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 568.231606] LNet: 60516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 568.234731] LNetError: 60516:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 568.247314] LNet: Removed LNI 192.168.202.53@tcp [ 568.503828] Key type .llcrypt unregistered [ 568.504920] Key type ._llcrypt unregistered [ 576.828376] Key type ._llcrypt registered [ 576.830181] Key type .llcrypt registered [ 577.525930] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 16:31:07 (1779309067) [ 577.963756] Key type .llcrypt unregistered [ 577.965291] Key type ._llcrypt unregistered [ 580.496808] Key type ._llcrypt registered [ 580.498706] Key type .llcrypt registered [ 580.608282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 580.615784] alg: No test for adler32 (adler32-zlib) [ 581.476635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 581.507527] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 585.189921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.153@tcp [ 588.646934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.675702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 588.689762] LNet: Added LNI 192.168.202.53@tcp1 [8/256/0/180] [ 589.503187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e remote_dropped -r 1 [ 589.544649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp1 -m GET -e remote_dropped -r 1 [ 589.568476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e remote_dropped -r 1 [ 589.592522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -m GET -e remote_dropped -r 1 [ 589.617599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp -m GET -e remote_dropped -r 1 [ 589.641943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp1 -m GET -e remote_dropped -r 1 [ 589.669150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -m GET -e remote_dropped -r 1 [ 589.696615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -m GET -e remote_dropped -r 1 [ 589.720487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 589.731362] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 589.735385] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.153@tcp [ 589.842675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 589.868268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 589.889792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 589.906093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 589.929596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 589.948499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 589.966700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 589.983753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 589.995158] LNet: 61978:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 590.007474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 590.024877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 590.041555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 590.065373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 590.084733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 590.105632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 590.127403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 590.154605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 590.423966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e remote_error -r 1 [ 590.442724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp1 -m GET -e remote_error -r 1 [ 590.466859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e remote_error -r 1 [ 590.491559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -m GET -e remote_error -r 1 [ 590.514456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp -m GET -e remote_error -r 1 [ 590.538745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp1 -m GET -e remote_error -r 1 [ 590.562397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -m GET -e remote_error -r 1 [ 590.589889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -m GET -e remote_error -r 1 [ 590.608429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 590.618449] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 590.622382] LNet: Skipped 2 previous similar messages [ 590.742035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 590.770237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 590.787365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 590.809792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 590.831223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 590.850486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 590.868638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 590.890739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 590.900313] LNet: 62101:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 590.910773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 590.936850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 590.961669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 590.980459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 591.001724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 591.022856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 591.047137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 591.074369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 591.363547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e remote_timeout -r 1 [ 591.384865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp1 -m GET -e remote_timeout -r 1 [ 591.408758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e remote_timeout -r 1 [ 591.435395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -m GET -e remote_timeout -r 1 [ 591.454554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp -m GET -e remote_timeout -r 1 [ 591.474711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp1 -m GET -e remote_timeout -r 1 [ 591.495746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -m GET -e remote_timeout -r 1 [ 591.517768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -m GET -e remote_timeout -r 1 [ 591.541285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.153@tcp [ 591.667709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 591.688962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 591.706767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 591.722935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 591.745586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 591.763903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 591.783814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 591.801797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 591.818223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 591.832542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 591.848749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 591.868389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 591.882488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 591.905819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 591.926760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 591.964087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 594.436300] LNet: 62301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 594.443607] LNetError: 62301:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.448864] LNet: Removed LNI 192.168.202.53@tcp [ 594.464490] LNet: Removed LNI 192.168.202.53@tcp1 [ 594.720808] Key type .llcrypt unregistered [ 594.722492] Key type ._llcrypt unregistered [ 602.183479] Key type ._llcrypt registered [ 602.184612] Key type .llcrypt registered [ 602.849416] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 16:31:32 (1779309092) [ 603.730976] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 604.580154] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 16:31:34 (1779309094) [ 605.032418] Key type .llcrypt unregistered [ 605.034098] Key type ._llcrypt unregistered [ 608.012410] Key type ._llcrypt registered [ 608.017523] Key type .llcrypt registered [ 608.129619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 608.138958] alg: No test for adler32 (adler32-zlib) [ 608.955141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 608.969109] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 612.580127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.153@tcp [ 612.911799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e network_timeout -r 1 [ 612.937620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e network_timeout -r 1 [ 612.959289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.153@tcp [ 612.970568] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 613.086774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 613.091254] LNet: 1 local NIs in recovery (showing 1): 192.168.202.53@tcp [ 613.105724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 613.123060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 613.138943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 613.156926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 613.174202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 613.196063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 613.218213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 613.230714] LNet: 63841:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 613.242721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 613.261622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 613.290977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 613.309573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 613.329724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 613.351320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 613.488376] LNet: 63880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 613.493944] LNetError: 63880:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 613.503932] LNet: Removed LNI 192.168.202.53@tcp [ 613.688411] Key type .llcrypt unregistered [ 613.691450] Key type ._llcrypt unregistered [ 621.460804] Key type ._llcrypt registered [ 621.462413] Key type .llcrypt registered [ 621.570788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 621.578757] alg: No test for adler32 (adler32-zlib) [ 622.470245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 622.497905] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 626.411885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.153@tcp [ 631.193704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.220753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 631.235449] LNet: Added LNI 192.168.202.53@tcp1 [8/256/0/180] [ 631.957563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e network_timeout -r 1 [ 631.989502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp1 -m GET -e network_timeout -r 1 [ 632.025975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e network_timeout -r 1 [ 632.060798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -m GET -e network_timeout -r 1 [ 632.095216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp -m GET -e network_timeout -r 1 [ 632.121922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp1 -m GET -e network_timeout -r 1 [ 632.145908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -m GET -e network_timeout -r 1 [ 632.163608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -m GET -e network_timeout -r 1 [ 632.182520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.153@tcp [ 632.194348] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 632.300502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 632.328866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 632.350570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 632.373485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 632.395346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 632.420509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 632.447688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 632.479635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 632.489857] LNet: 64756:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 632.500219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 632.527293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 632.546866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 632.567261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 632.590498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 632.612596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 635.199217] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 635.343660] LNet: 64829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 635.347911] LNetError: 64829:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.383168] LNet: Removed LNI 192.168.202.53@tcp [ 636.385192] LNetError: 64829:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.403922] LNet: Removed LNI 192.168.202.53@tcp1 [ 636.657423] Key type .llcrypt unregistered [ 636.660855] Key type ._llcrypt unregistered [ 644.327843] Key type ._llcrypt registered [ 644.329248] Key type .llcrypt registered [ 644.942989] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 16:32:14 (1779309134) [ 645.171333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 645.180296] alg: No test for adler32 (adler32-zlib) [ 646.016319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.050655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 646.064477] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 646.426397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 646.442230] LNet: Added LNI 192.168.202.53@tcp1 [8/256/0/180] [ 646.871343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 646.919063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 646.941840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 646.963677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 646.983879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 647.006485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 647.026128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.53@tcp [ 647.037971] LNetError: 65927:0:(api-ni.c:4351:lnet_ni_set_healthv()) manually adding local NI 192.168.202.53@tcp to recovery [ 647.072898] LNet: 1 local NIs in recovery (showing 1): 192.168.202.53@tcp [ 647.075894] LNet: There was an unexpected network error while writing to 192.168.202.53: rc = -22 [ 649.119871] LNet: There was an unexpected network error while writing to 192.168.202.53: rc = -22 [ 649.245897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 653.216057] LNet: There was an unexpected network error while writing to 192.168.202.53: rc = -22 [ 653.548303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 653.575469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 653.611772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.625518] LNet: 66034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.636106] LNetError: 66034:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 653.640297] LNet: Removed LNI 192.168.202.53@tcp [ 653.652968] LNet: Removed LNI 192.168.202.53@tcp1 [ 653.664846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.695453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 653.711411] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 654.056243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 654.070748] LNet: Added LNI 192.168.202.53@tcp1 [8/256/0/180] [ 654.503967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 654.525603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 654.555510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 654.578636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 654.598214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 654.615843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 654.633789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 654.648696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.53@tcp [ 654.659677] LNetError: 66306:0:(api-ni.c:4351:lnet_ni_set_healthv()) manually adding local NI 192.168.202.53@tcp to recovery [ 656.735332] LNet: There was an unexpected network error while writing to 192.168.202.53: rc = -22 [ 656.738564] LNet: Skipped 1 previous similar message [ 656.858854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 660.831795] LNet: There was an unexpected network error while writing to 192.168.202.53: rc = -22 [ 665.347729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 665.361447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 665.378212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 668.491400] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 16:32:38 (1779309158) [ 668.536710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.554188] LNet: 67015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.559821] LNetError: 67015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.563707] LNetError: 67015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 669.601513] LNet: Removed LNI 192.168.202.53@tcp [ 669.603703] LNetError: 67015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 669.617790] LNet: Removed LNI 192.168.202.53@tcp1 [ 669.636288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.675327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 669.693853] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 670.078339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 670.090552] LNet: Added LNI 192.168.202.53@tcp1 [8/256/0/180] [ 670.563564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 670.597557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 670.612561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 670.630916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 670.651272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 670.677679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 670.694339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.53@tcp --health 0 [ 670.753245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 671.711258] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.53@tcp [ 671.711695] LNet: There was an unexpected network error while writing to 192.168.202.53: rc = -22 [ 671.720630] LNet: Skipped 1 previous similar message [ 685.430542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 685.505520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 685.526104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 685.547076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.53@tcp --health 500 [ 685.602338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 691.969944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 691.996670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.011147] LNet: 67514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.015261] LNetError: 67514:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.023161] LNet: Removed LNI 192.168.202.53@tcp [ 693.034427] LNet: Removed LNI 192.168.202.53@tcp1 [ 693.043583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.067812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 693.079633] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 693.422510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 693.433931] LNet: Added LNI 192.168.202.53@tcp1 [8/256/0/180] [ 693.827391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 693.848650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 693.881035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 693.896605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 693.918278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 693.936949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 693.953473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 693.970571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.53@tcp --health 0 [ 694.111802] LNet: There was an unexpected network error while writing to 192.168.202.53: rc = -22 [ 694.116604] LNet: Skipped 3 previous similar messages [ 695.154081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 704.682980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 704.700741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 704.720275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 707.885449] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 16:33:17 (1779309197) [ 708.185629] LNet: 68560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 708.189692] LNetError: 68560:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 708.194366] LNetError: 68560:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 708.202534] LNet: Removed LNI 192.168.202.53@tcp [ 708.212524] LNet: Removed LNI 192.168.202.53@tcp1 [ 708.509975] Key type .llcrypt unregistered [ 708.511632] Key type ._llcrypt unregistered [ 711.322867] Key type ._llcrypt registered [ 711.324360] Key type .llcrypt registered [ 711.432561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 711.439614] alg: No test for adler32 (adler32-zlib) [ 712.273639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 712.292521] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 716.695117] Lustre: DEBUG MARKER: Initial discovery [ 716.717350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.153@tcp [ 717.750457] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 717.833529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 717.864504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.153@tcp [ 717.877196] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 717.891208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 717.918342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.153@tcp -r 1 -m GET -l 3 [ 717.949548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.153@tcp -r 1 -m GET -e local_error [ 717.970600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.153@tcp -r 1 -m PUT -l 6 [ 718.432985] LNet: 1 local NIs in recovery (showing 1): 192.168.202.53@tcp [ 718.703370] Lustre: DEBUG MARKER: Force oleg253-client.virtnet to discover 192.168.202.153@tcp (in background) [ 718.731761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.153@tcp [ 719.357128] Lustre: DEBUG MARKER: Force oleg253-server to discover 192.168.202.53@tcp [ 720.409950] Lustre: DEBUG MARKER: Wait for 69314 [ 721.759477] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 722.385291] Lustre: DEBUG MARKER: Finished wait on 69314 [ 722.433715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 722.451557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 722.573365] LNet: 69770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.576798] LNetError: 69770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.590448] LNet: Removed LNI 192.168.202.53@tcp [ 722.744637] Key type .llcrypt unregistered [ 722.747320] Key type ._llcrypt unregistered [ 728.672255] Key type ._llcrypt registered [ 728.674071] Key type .llcrypt registered [ 729.216176] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 16:33:39 (1779309219) [ 729.479449] Key type .llcrypt unregistered [ 729.481456] Key type ._llcrypt unregistered [ 731.482396] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 731.542547] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 731.549178] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 731.552412] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 733.756339] Key type ._llcrypt registered [ 733.757889] Key type .llcrypt registered [ 733.826407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 733.833446] alg: No test for adler32 (adler32-zlib) [ 734.643196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.664604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 734.674921] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 734.944523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 734.954849] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 738.372620] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 16:33:48 (1779309228) [ 738.410587] LNet: 71838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.413719] LNetError: 71838:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 738.416514] LNet: Removed LNI 192.168.202.53@tcp [ 738.426706] LNet: Removed LNI 10.1.2.3@tcp [ 738.698439] Key type .llcrypt unregistered [ 738.699806] Key type ._llcrypt unregistered [ 741.073269] Key type ._llcrypt registered [ 741.074370] Key type .llcrypt registered [ 741.149158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 741.155223] alg: No test for adler32 (adler32-zlib) [ 741.969615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.998887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 742.011507] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 742.303148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 742.312843] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 742.681751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.53@tcp [ 743.711142] LNetError: Unexpected error -22 connecting to 192.168.202.53@tcp at host 192.168.202.53:988 [ 744.985818] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 16:33:54 (1779309234) [ 745.016613] LNet: 72968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.019377] LNetError: 72968:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.022320] LNet: Removed LNI 192.168.202.53@tcp [ 745.030046] LNet: Removed LNI 10.1.2.3@tcp [ 745.284425] Key type .llcrypt unregistered [ 745.285408] Key type ._llcrypt unregistered [ 747.287290] Key type ._llcrypt registered [ 747.288777] Key type .llcrypt registered [ 747.352595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 747.359278] alg: No test for adler32 (adler32-zlib) [ 748.165885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.187443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 748.198816] LNet: Added LNI 192.168.202.53@tcp1 [8/256/0/180] [ 748.468354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 748.476976] LNet: Added LNI 192.168.202.53@tcp2 [8/256/0/180] [ 748.814468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.53@tcp1 --nid 192.168.202.53@tcp2 [ 750.277886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.53@tcp1 192.168.202.53@tcp2 [ 750.286944] LNetError: 73810:0:(lib-move.c:2021:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.53@tcp1 to 192.168.202.53@tcp2 - Route table may be misconfigured [ 750.295779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.53@tcp2 192.168.202.53@tcp1 [ 752.491656] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 16:34:02 (1779309242) [ 752.545300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.556586] LNet: 74381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.559937] LNetError: 74381:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 752.563105] LNet: Removed LNI 192.168.202.53@tcp1 [ 753.573921] LNet: Removed LNI 192.168.202.53@tcp2 [ 753.582983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.606062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 753.616830] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 753.884363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 753.893996] LNet: Added LNI 192.168.202.53@tcp1 [8/256/0/180] [ 754.212811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 754.229844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.53@tcp 192.168.202.53@tcp [ 754.245452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.53@tcp1 192.168.202.53@tcp1 [ 754.261508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.53@tcp -d 192.168.202.53@tcp -e network_timeout [ 754.276254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -e network_timeout [ 754.291736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -e network_timeout [ 754.308373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -e network_timeout [ 754.325028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 754.333312] LNet: There was an unexpected network error while writing to 192.168.202.53: rc = -22 [ 754.341310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 754.377336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 754.414405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 754.656103] LNet: 1 local NIs in recovery (showing 1): 192.168.202.53@tcp [ 756.604759] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 16:34:06 (1779309246) [ 756.646288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 756.659274] LNet: 75237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 756.662879] LNetError: 75237:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 756.666130] LNetError: 75237:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 756.670347] LNet: Removed LNI 192.168.202.53@tcp [ 757.733477] LNet: Removed LNI 192.168.202.53@tcp1 [ 757.742200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.771033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 757.859886] LNet: 75280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.007757] Key type .llcrypt unregistered [ 758.009198] Key type ._llcrypt unregistered [ 761.945047] Key type ._llcrypt registered [ 761.946489] Key type .llcrypt registered [ 762.432793] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 16:34:12 (1779309252) [ 762.602144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 762.608932] alg: No test for adler32 (adler32-zlib) [ 763.416811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.974086] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 764.658793] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 16:34:14 (1779309254) [ 764.694994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 764.707074] LNet: 76371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.720805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.754750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 764.769355] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 765.068345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 765.078595] LNet: Added LNI 192.168.202.53@tcp1 [8/256/0/180] [ 765.408133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 765.423337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp1 [ 765.438481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp1 [ 767.892961] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 16:34:17 (1779309257) [ 768.645726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg253-server [ 769.271935] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 16:34:19 (1779309259) [ 769.980985] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg253-server [ 770.629733] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 16:34:20 (1779309260) [ 771.344614] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg253-server [ 772.005842] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 16:34:21 (1779309261) [ 772.707404] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg253-server [ 773.395103] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 16:34:23 (1779309263) [ 774.177961] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg253-server [ 774.877567] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 16:34:24 (1779309264) [ 775.647988] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg253-server [ 776.321479] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 16:34:26 (1779309266) [ 777.134790] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg253-server [ 777.863866] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 16:34:27 (1779309267) [ 778.754276] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg253-server [ 779.570692] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 16:34:29 (1779309269) [ 780.455814] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg253-server [ 781.299977] Lustre: DEBUG MARKER: == sanity-lnet test 229: Check NI status with lnetctl net disable/enable ========================================================== 16:34:31 (1779309271) [ 781.342708] LNet: 80241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.347362] LNetError: 80241:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.350155] LNet: Removed LNI 192.168.202.53@tcp [ 782.373409] LNet: Removed LNI 192.168.202.53@tcp1 [ 782.686491] Key type .llcrypt unregistered [ 782.687552] Key type ._llcrypt unregistered [ 784.813216] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 784.815158] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 787.021366] Key type ._llcrypt registered [ 787.024624] Key type .llcrypt registered [ 787.115021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 787.122094] alg: No test for adler32 (adler32-zlib) [ 787.973521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.015248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 788.032427] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 788.377758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 788.392343] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 789.390395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp --if test1pg [ 789.444504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 789.467589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net enable --net tcp --if test1pg [ 789.515802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp --if ens2 [ 789.573461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 10.1.2.3@tcp [ 789.592022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net enable --net tcp --if ens2 [ 789.640814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp [ 789.658523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --if ens2 [ 789.832131] LNet: 81063:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.835318] LNetError: 81063:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 789.839120] LNet: Removed LNI 192.168.202.53@tcp [ 790.884363] LNet: Removed LNI 10.1.2.3@tcp [ 791.067279] Key type .llcrypt unregistered [ 791.068939] Key type ._llcrypt unregistered [ 795.377100] Key type ._llcrypt registered [ 795.377966] Key type .llcrypt registered [ 795.976187] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 16:34:45 (1779309285) [ 796.140697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 796.150555] alg: No test for adler32 (adler32-zlib) [ 796.962607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.983786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 796.994231] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 797.324630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 797.369664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 797.472588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.481656] LNet: 82003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.484719] LNetError: 82003:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.497339] LNet: Removed LNI 192.168.202.53@tcp [ 797.506656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.533504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 797.546431] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 797.914721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 798.002954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.014806] LNet: 82148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.018312] LNetError: 82148:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.032450] LNet: Removed LNI 192.168.202.53@tcp [ 798.046232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.074533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 798.088439] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 798.426664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 798.466805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 798.549814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.592610] LNet: Removed LNI 192.168.202.53@tcp [ 799.602109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.634635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 799.651126] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 799.997894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 800.083148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.094458] LNet: 82440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.097144] LNet: 82440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 800.100183] LNetError: 82440:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.104768] LNetError: 82440:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 800.117614] LNet: Removed LNI 192.168.202.53@tcp [ 800.129458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.155697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 800.172698] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 800.495651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 800.534291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 800.637255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.703144] LNet: Removed LNI 192.168.202.53@tcp [ 801.713984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.745895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 801.765955] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 802.177471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 802.271470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.286537] LNet: 82733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.289228] LNet: 82733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 802.291811] LNetError: 82733:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.294275] LNetError: 82733:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 802.301707] LNet: Removed LNI 192.168.202.53@tcp [ 802.317389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.345522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 802.357059] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 802.642202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 802.676665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 802.772658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.813337] LNet: Removed LNI 192.168.202.53@tcp [ 803.822418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.847327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 803.860292] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 804.209829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 804.300917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.349816] LNet: Removed LNI 192.168.202.53@tcp [ 805.358504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.380949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 805.393794] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 805.682164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 805.711744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 806.866908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.877806] LNet: 83181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.881120] LNet: 83181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 806.884427] LNetError: 83181:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.887973] LNetError: 83181:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 806.896601] LNet: Removed LNI 192.168.202.53@tcp [ 806.907403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.934571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 806.946152] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 807.256344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 807.350632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.421396] LNet: Removed LNI 192.168.202.53@tcp [ 808.429855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.455989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 808.468692] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 808.782109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 808.814180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 808.905234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.957446] LNet: Removed LNI 192.168.202.53@tcp [ 809.965968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.993935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 810.004206] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 810.346365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 810.442466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.493350] LNet: Removed LNI 192.168.202.53@tcp [ 811.502256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.525317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.536962] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 811.847209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 811.877758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.893346] LNet: Removed LNI 192.168.202.53@tcp [ 811.901320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.924465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.932453] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 812.228889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 814.748434] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 16:35:04 (1779309304) [ 814.789332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.814551] LNet: Removed LNI 192.168.202.53@tcp [ 814.834192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.866558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 814.880411] LNet: Added LNI 192.168.202.53@tcp231 [8/256/0/180] [ 815.373707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.383085] LNet: 84593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.386689] LNet: 84593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 815.390085] LNetError: 84593:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.392782] LNetError: 84593:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 815.399617] LNet: Removed LNI 192.168.202.53@tcp231 [ 815.408092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.436661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 815.448396] LNet: Added LNI 192.168.202.53@tcp231 [8/256/0/180] [ 815.451185] LNet: 84601:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 815.631700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 815.649418] LNet: Removed LNI 192.168.202.53@tcp231 [ 815.657132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 815.667672] LNet: Added LNI 192.168.202.53@tcp231 [8/256/0/180] [ 816.161161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.189637] LNet: Removed LNI 192.168.202.53@tcp231 [ 816.198767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.232374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 816.249777] LNet: Added LNI 192.168.202.53@tcp231 [8/256/0/0] [ 816.252414] LNet: 84753:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 816.421640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 816.438714] LNet: Removed LNI 192.168.202.53@tcp231 [ 816.445798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 816.458891] LNet: Added LNI 192.168.202.53@tcp231 [8/256/0/0] [ 816.923866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.936784] LNet: Removed LNI 192.168.202.53@tcp231 [ 816.946233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.979077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 816.994624] LNet: Added LNI 192.168.202.53@tcp231 [8/256/0/60] [ 817.156906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 817.179387] LNet: Removed LNI 192.168.202.53@tcp231 [ 817.186290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 817.197266] LNet: Added LNI 192.168.202.53@tcp231 [8/256/0/60] [ 817.656525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.675283] LNet: Removed LNI 192.168.202.53@tcp231 [ 817.687698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.722038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 817.736191] LNet: Added LNI 192.168.202.53@tcp231 [8/256/0/180] [ 817.740615] LNet: 85055:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 817.744046] LNet: 85055:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 817.893707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 817.916596] LNet: Removed LNI 192.168.202.53@tcp231 [ 817.924562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 817.937373] LNet: Added LNI 192.168.202.53@tcp231 [8/256/0/180] [ 820.637559] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 16:35:10 (1779309310) [ 820.682781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.700387] LNet: Removed LNI 192.168.202.53@tcp231 [ 820.710916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.737849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 820.748578] LNet: Added LNI 192.168.202.53@tcp232 [8/256/0/180] [ 821.032883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 821.047279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 821.066580] LNet: Removed LNI 192.168.202.53@tcp232 [ 821.073898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 821.085248] LNet: Added LNI 192.168.202.53@tcp232 [8/256/0/180] [ 821.342974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 821.359017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 821.375627] LNet: Removed LNI 192.168.202.53@tcp232 [ 821.384093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 821.393920] LNet: Added LNI 192.168.202.53@tcp232 [8/256/0/180] [ 821.680549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 821.699777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 821.720380] LNet: Removed LNI 192.168.202.53@tcp232 [ 821.727266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 821.736431] LNet: Added LNI 192.168.202.53@tcp232 [8/256/0/180] [ 822.055832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 822.073553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 822.090487] LNet: Removed LNI 192.168.202.53@tcp232 [ 822.096513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 824.255657] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 16:35:14 (1779309314) [ 824.624821] Key type .llcrypt unregistered [ 824.626348] Key type ._llcrypt unregistered [ 826.781776] Key type ._llcrypt registered [ 826.783388] Key type .llcrypt registered [ 826.861607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 826.868946] alg: No test for adler32 (adler32-zlib) [ 827.678726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 827.690804] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 830.689191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.153@tcp [ 833.848696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.867210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 833.883364] LNet: Added LNI 192.168.202.53@tcp1 [8/256/0/180] [ 834.534904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_interrupt -r 2 [ 834.552733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp1 -m GET -e local_interrupt -r 2 [ 834.572960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_interrupt -r 2 [ 834.595031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -m GET -e local_interrupt -r 2 [ 834.617840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp -m GET -e local_interrupt -r 2 [ 834.641592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp1 -m GET -e local_interrupt -r 2 [ 834.663745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -m GET -e local_interrupt -r 2 [ 834.683644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -m GET -e local_interrupt -r 2 [ 834.705741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.53@tcp 192.168.202.153@tcp [ 834.717248] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 834.720148] LNet: 1 local NIs in recovery (showing 1): 192.168.202.53@tcp [ 834.813708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 834.829378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 834.843995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 834.857528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 834.873326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 834.888328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 834.904632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 834.919937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 834.931491] LNet: 87546:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 834.941413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 834.962806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 834.978079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 834.993177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 835.009840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 835.027893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 835.235790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_dropped -r 2 [ 835.256035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp1 -m GET -e local_dropped -r 2 [ 835.271357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_dropped -r 2 [ 835.290497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -m GET -e local_dropped -r 2 [ 835.307860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp -m GET -e local_dropped -r 2 [ 835.324843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp1 -m GET -e local_dropped -r 2 [ 835.344140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -m GET -e local_dropped -r 2 [ 835.360606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -m GET -e local_dropped -r 2 [ 835.380156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.53@tcp 192.168.202.153@tcp [ 835.390906] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 835.494395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 835.509549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 835.523770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 835.538905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 835.556076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 835.571284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 835.588335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 835.604367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 835.611818] LNet: 87666:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 835.620764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 835.635957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 835.657988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 835.672548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 835.689325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 835.708396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 835.923422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_aborted -r 2 [ 835.941749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp1 -m GET -e local_aborted -r 2 [ 835.959221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_aborted -r 2 [ 835.977266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -m GET -e local_aborted -r 2 [ 835.994885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp -m GET -e local_aborted -r 2 [ 836.013269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp1 -m GET -e local_aborted -r 2 [ 836.032693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -m GET -e local_aborted -r 2 [ 836.051853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -m GET -e local_aborted -r 2 [ 836.067652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.53@tcp 192.168.202.153@tcp [ 836.181915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 836.197269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 836.213591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 836.228890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 836.246967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 836.261920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 836.277792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 836.292041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 836.308657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 836.323177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 836.340473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 836.354400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 836.368822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 836.382480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 836.599809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_no_route -r 2 [ 836.618695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp1 -m GET -e local_no_route -r 2 [ 836.637237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_no_route -r 2 [ 836.660688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -m GET -e local_no_route -r 2 [ 836.680815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp -m GET -e local_no_route -r 2 [ 836.702620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp1 -m GET -e local_no_route -r 2 [ 836.724223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -m GET -e local_no_route -r 2 [ 836.742801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -m GET -e local_no_route -r 2 [ 836.765737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.53@tcp 192.168.202.153@tcp [ 836.777088] LNet: There was an unexpected network error while writing to 192.168.202.153: rc = -22 [ 836.780416] LNet: Skipped 2 previous similar messages [ 836.884441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 836.903559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 836.922136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 836.938565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 836.953278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 836.970245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 836.986488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 837.002927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 837.013490] LNet: 87906:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 837.016188] LNet: 87906:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 837.026083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 837.042829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 837.059870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 837.077401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 837.093166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 837.108768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 837.353689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e local_timeout -r 2 [ 837.371143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp1 -m GET -e local_timeout -r 2 [ 837.387367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e local_timeout -r 2 [ 837.405463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -m GET -e local_timeout -r 2 [ 837.423850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp -m GET -e local_timeout -r 2 [ 837.440988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp1 -m GET -e local_timeout -r 2 [ 837.461361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -m GET -e local_timeout -r 2 [ 837.482883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -m GET -e local_timeout -r 2 [ 837.508430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.53@tcp 192.168.202.153@tcp [ 837.620623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 837.638139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 837.657353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 837.673230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 837.689220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 837.705710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 837.722483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 837.745907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 837.766858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 837.785220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 837.801019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 837.822872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 837.841276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 837.865860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 838.074617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp -m GET -e remote_dropped -r 2 [ 838.091792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.153@tcp1 -m GET -e remote_dropped -r 2 [ 838.112939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp -m GET -e remote_dropped -r 2 [ 838.131480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp -d 192.168.202.53@tcp1 -m GET -e remote_dropped -r 2 [ 838.148777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp -m GET -e remote_dropped -r 2 [ 838.165729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.153@tcp1 -m GET -e remote_dropped -r 2 [ 838.182278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp -m GET -e remote_dropped -r 2 [ 838.197590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.53@tcp1 -d 192.168.202.53@tcp1 -m GET -e remote_dropped -r 2 [ 838.216834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.53@tcp 192.168.202.153@tcp [ 838.229453] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.153@tcp [ 838.316312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 838.328200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 838.341535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 838.353834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 838.367123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 838.380147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 838.395268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 838.409475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 838.424957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 838.439701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 838.453279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 838.468314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 838.481911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 838.496629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 840.803753] LNet: 88220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.807322] LNetError: 88220:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.811184] LNet: Removed LNI 192.168.202.53@tcp [ 840.820481] LNet: Removed LNI 192.168.202.53@tcp1 [ 840.966750] Key type .llcrypt unregistered [ 840.968202] Key type ._llcrypt unregistered [ 846.499115] Key type ._llcrypt registered [ 846.500621] Key type .llcrypt registered [ 846.997755] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 16:35:36 (1779309336) [ 847.130900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 847.136828] alg: No test for adler32 (adler32-zlib) [ 847.947633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.972965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 847.985894] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 848.282746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 848.296882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.53@tcp [ 848.307398] LNetError: 89190:0:(api-ni.c:4351:lnet_ni_set_healthv()) manually adding local NI 192.168.202.53@tcp to recovery [ 848.345147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.53@tcp [ 848.383082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 850.718754] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 16:35:40 (1779309340) [ 850.758377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 850.768736] LNet: 89774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.771370] LNetError: 89774:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.780581] LNet: Removed LNI 192.168.202.53@tcp [ 850.790677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.810565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 850.819459] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 851.117284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 851.127561] LNet: Added LNI 192.168.202.53@tcp1 [8/256/0/180] [ 851.476735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.53@tcp [ 851.485374] LNetError: 90044:0:(api-ni.c:4351:lnet_ni_set_healthv()) manually adding local NI 192.168.202.53@tcp to recovery [ 851.807424] LNet: 1 local NIs in recovery (showing 1): 192.168.202.53@tcp [ 851.807954] LNet: There was an unexpected network error while writing to 192.168.202.53: rc = -22 [ 852.573344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.53@tcp [ 856.854673] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 16:35:46 (1779309346) [ 857.476166] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg253-server [ 858.117545] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 16:35:48 (1779309348) [ 858.782397] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg253-server [ 859.380450] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 16:35:49 (1779309349) [ 860.001254] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg253-server [ 860.580571] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 16:35:50 (1779309350) [ 860.611572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.622343] LNet: 91667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.625434] LNetError: 91667:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.627763] LNet: Removed LNI 192.168.202.53@tcp [ 860.635705] LNet: Removed LNI 192.168.202.53@tcp1 [ 860.646440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.676223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 860.688315] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 860.984655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 861.015372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 861.037117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 861.067578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 861.089263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 861.108829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 863.379259] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 16:35:53 (1779309353) [ 863.427073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.436753] LNet: 92393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.441254] LNetError: 92393:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.445894] LNetError: 92393:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 863.454739] LNet: Removed LNI 192.168.202.53@tcp [ 863.465299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.491356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 863.506266] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 866.147991] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 16:35:56 (1779309356) [ 866.180423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.191437] LNet: 93094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.195438] LNetError: 93094:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.205859] LNet: Removed LNI 192.168.202.53@tcp [ 866.215209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.244555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 866.257235] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 868.903307] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 16:35:58 (1779309358) [ 869.478522] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 870.200658] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 16:36:00 (1779309360) [ 870.364697] LNet: 94134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.367286] LNetError: 94134:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.374482] LNet: Removed LNI 192.168.202.53@tcp [ 870.633786] Key type .llcrypt unregistered [ 870.635690] Key type ._llcrypt unregistered [ 872.749429] Key type ._llcrypt registered [ 872.750910] Key type .llcrypt registered [ 872.824652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 872.831292] alg: No test for adler32 (adler32-zlib) [ 873.646235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 873.657537] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 876.891910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.153@tcp [ 878.588260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.153@tcp [ 878.733702] LNet: 94543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.737270] LNetError: 94543:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.747637] LNet: Removed LNI 192.168.202.53@tcp [ 878.934452] Key type .llcrypt unregistered [ 878.936112] Key type ._llcrypt unregistered [ 883.203373] Key type ._llcrypt registered [ 883.205017] Key type .llcrypt registered [ 883.748085] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 884.340580] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 885.028234] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 16:36:14 (1779309374) [ 885.622982] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 886.246877] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 16:36:16 (1779309376) [ 886.935540] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg253-server [ 887.582448] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 16:36:17 (1779309377) [ 888.297719] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg253-server [ 888.946213] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 16:36:18 (1779309378) [ 889.459741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 889.467467] alg: No test for adler32 (adler32-zlib) [ 890.332977] Lustre: Lustre: Build Version: 2.17.53_24_ge4eab26 [ 890.416262] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 892.007113] Key type lgssc registered [ 892.410159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 900.743888] Key type lgssc unregistered [ 900.868421] LNet: 97699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.871215] LNetError: 97699:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.879412] LNet: Removed LNI 192.168.202.53@tcp [ 901.143691] Key type .llcrypt unregistered [ 901.145217] Key type ._llcrypt unregistered [ 906.808770] Key type ._llcrypt registered [ 906.810217] Key type .llcrypt registered [ 907.303912] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 16:36:37 (1779309397) [ 907.461887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 907.468528] alg: No test for adler32 (adler32-zlib) [ 908.287410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.317420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 908.338119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 908.358840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 908.381299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 910.983685] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 16:36:40 (1779309400) [ 911.018385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.027969] LNet: 99102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.042960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.074544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 911.087518] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 911.471255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.481269] LNetError: 99252:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.488371] LNet: Removed LNI 192.168.202.53@tcp [ 911.498898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.526709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 911.538763] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 911.727208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.737069] LNet: 99280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.740068] LNet: 99280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 911.749622] LNet: Removed LNI 192.168.202.53@tcp [ 911.758822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.780651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 911.792183] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 914.454356] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 16:36:44 (1779309404) [ 914.486651] LNet: 99863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.490659] LNetError: 99863:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.493928] LNetError: 99863:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 914.507402] LNet: Removed LNI 192.168.202.53@tcp [ 914.748894] Key type .llcrypt unregistered [ 914.751266] Key type ._llcrypt unregistered [ 916.913185] Key type ._llcrypt registered [ 916.914516] Key type .llcrypt registered [ 917.015952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 917.022140] alg: No test for adler32 (adler32-zlib) [ 917.830485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.864165] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 917.866712] LNet: *** cfs_fail_loc=e003, val=100*** [ 918.162178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 918.172272] Ping reply received with mapping: [ 918.173801] found metadata reply NI[192.168.202.53@tcp] [ 918.175863] 35 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 918.177555] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 918.179387] 5a 5a 5a 5a 5a 5a 5a 5a [ 918.237931] LNet: 100261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.241480] LNetError: 100261:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.249359] LNet: Removed LNI 192.168.202.53@tcp [ 918.482391] Key type .llcrypt unregistered [ 918.484363] Key type ._llcrypt unregistered [ 922.697337] Key type ._llcrypt registered [ 922.699266] Key type .llcrypt registered [ 923.246726] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 16:36:53 (1779309413) [ 923.458277] Key type .llcrypt unregistered [ 923.460381] Key type ._llcrypt unregistered [ 925.842267] Key type ._llcrypt registered [ 925.843459] Key type .llcrypt registered [ 925.945391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 925.952254] alg: No test for adler32 (adler32-zlib) [ 926.888683] LNetError: 101292:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 927.066471] Key type .llcrypt unregistered [ 927.068761] Key type ._llcrypt unregistered [ 929.248333] Key type ._llcrypt registered [ 929.249441] Key type .llcrypt registered [ 929.884874] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 16:36:59 (1779309419) [ 930.101340] Key type .llcrypt unregistered [ 930.102772] Key type ._llcrypt unregistered [ 932.396904] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 932.451053] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 932.453275] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 932.455885] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 934.706834] Key type ._llcrypt registered [ 934.708012] Key type .llcrypt registered [ 934.796733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 934.807850] alg: No test for adler32 (adler32-zlib) [ 935.625150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.656760] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 939.920423] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 16:37:09 (1779309429) [ 939.958167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 939.969838] LNet: 103077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.973410] LNetError: 103077:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 939.981605] LNet: Removed LNI 10.1.2.3@tcp [ 939.996416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.307434] Key type .llcrypt unregistered [ 940.309283] Key type ._llcrypt unregistered [ 942.517684] Key type ._llcrypt registered [ 942.518845] Key type .llcrypt registered [ 942.595106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 942.603128] alg: No test for adler32 (adler32-zlib) [ 943.434159] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 943.461505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 943.536135] LNet: 103370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.539975] LNetError: 103370:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.547456] LNet: Removed LNI 192.168.202.53@tcp [ 943.853454] Key type .llcrypt unregistered [ 943.855598] Key type ._llcrypt unregistered [ 947.987201] Key type ._llcrypt registered [ 947.988431] Key type .llcrypt registered [ 948.513608] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 16:37:18 (1779309438) [ 948.733757] Key type .llcrypt unregistered [ 948.735327] Key type ._llcrypt unregistered [ 950.855179] Key type ._llcrypt registered [ 950.856919] Key type .llcrypt registered [ 950.934889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 950.941133] alg: No test for adler32 (adler32-zlib) [ 954.154842] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 16:37:24 (1779309444) [ 954.187816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.203814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.233563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 954.243636] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 954.524799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 954.539071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 954.555068] LNet: Removed LNI 192.168.202.53@tcp [ 954.654132] LNet: 105111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.805355] Key type .llcrypt unregistered [ 954.807309] Key type ._llcrypt unregistered [ 958.661138] Key type ._llcrypt registered [ 958.662754] Key type .llcrypt registered [ 959.198095] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 16:37:29 (1779309449) [ 959.332348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 959.339042] alg: No test for adler32 (adler32-zlib) [ 960.150395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.179378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 960.190933] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 960.486362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 960.502365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.53@tcp [ 960.511236] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 960.513322] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 960.514886] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 960, 0, 0 [ 960.516380] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 960.517253] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 960.518444] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 960.519508] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 960, 0, 0 [ 960.521039] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 960.522209] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 960.523580] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 960.524624] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 960, 0, 1010 [ 960.526268] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 960.527224] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 960.528529] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 960.529598] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 960, 0, 1010 [ 960.532024] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 962.795584] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 16:37:32 (1779309452) [ 962.997084] LNet: 106655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.000251] LNetError: 106655:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.010567] LNet: Removed LNI 192.168.202.53@tcp [ 963.275729] Key type .llcrypt unregistered [ 963.276839] Key type ._llcrypt unregistered [ 965.415474] Key type ._llcrypt registered [ 965.417014] Key type .llcrypt registered [ 965.490805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 965.497167] alg: No test for adler32 (adler32-zlib) [ 966.316488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 966.330804] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 969.333093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.153@tcp [ 972.378539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 972.398685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 972.408424] LNet: Added LNI 192.168.202.53@tcp1 [8/256/0/180] [ 972.822875] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 972.889872] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 972.892395] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 972.895640] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 975.071363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 975.081954] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 979.542794] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 16:37:49 (1779309469) [ 979.584344] LNet: 108154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.587528] LNetError: 108154:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.594710] LNet: Removed LNI 192.168.202.53@tcp [ 980.639244] LNet: Removed LNI 192.168.202.53@tcp1 [ 980.642144] LNetError: 108154:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 980.645859] LNetError: 108154:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 980.655834] LNet: Removed LNI 10.1.2.3@tcp99 [ 980.925543] Key type .llcrypt unregistered [ 980.927154] Key type ._llcrypt unregistered [ 983.355801] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 983.358645] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 985.567290] Key type ._llcrypt registered [ 985.568765] Key type .llcrypt registered [ 985.647701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 985.654217] alg: No test for adler32 (adler32-zlib) [ 986.472907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.496726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 986.506674] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 986.791066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 986.800710] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 987.688872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 987.712882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 987.735069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 987.746929] LNetError: 108903:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 987.761745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 987.780680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 987.947530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 987.977031] LNet: 108923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.980057] LNetError: 108923:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.983711] LNet: Removed LNI 192.168.202.53@tcp [ 987.992391] LNet: Removed LNI 10.1.2.3@tcp [ 988.236704] Key type .llcrypt unregistered [ 988.238835] Key type ._llcrypt unregistered [ 992.536404] Key type ._llcrypt registered [ 992.538028] Key type .llcrypt registered [ 993.120581] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 16:38:02 (1779309482) [ 993.281907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 993.288497] alg: No test for adler32 (adler32-zlib) [ 994.105521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.127628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 994.139898] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 994.447347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.53@tcp [ 994.476443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg253-client@tcp [ 996.982683] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 16:38:06 (1779309486) [ 997.017820] LNet: 110448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.020251] LNetError: 110448:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.028818] LNet: Removed LNI 192.168.202.53@tcp [ 997.292980] Key type .llcrypt unregistered [ 997.294795] Key type ._llcrypt unregistered [ 999.448832] Key type ._llcrypt registered [ 999.450238] Key type .llcrypt registered [ 999.516991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 999.524670] alg: No test for adler32 (adler32-zlib) [ 1000.324206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1000.334375] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1000.346321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1002.489937] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 16:38:12 (1779309492) [ 1003.034570] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1003.627858] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 16:38:13 (1779309493) [ 1004.155239] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1004.785657] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 16:38:14 (1779309494) [ 1004.836775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1004.845546] LNet: 111870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.848471] LNetError: 111870:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.856792] LNet: Removed LNI 192.168.202.53@tcp [ 1004.865121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.888210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1004.898750] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1005.180805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1005.199348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1007.702608] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 16:38:17 (1779309497) [ 1007.737191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.747416] LNet: 112574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.750978] LNetError: 112574:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.763702] LNet: Removed LNI 192.168.202.53@tcp [ 1007.778804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.803322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1007.816700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1008.045407] Key type .llcrypt unregistered [ 1008.046777] Key type ._llcrypt unregistered [ 1011.946240] Key type ._llcrypt registered [ 1011.947560] Key type .llcrypt registered [ 1012.444551] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 16:38:22 (1779309502) [ 1012.593604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1012.602251] alg: No test for adler32 (adler32-zlib) [ 1013.411115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.434883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1013.450105] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1013.725340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1013.746903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1016.154763] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 16:38:25 (1779309505) [ 1016.215156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1016.229879] LNet: 114133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.233339] LNetError: 114133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.244156] LNet: Removed LNI 192.168.202.53@tcp [ 1016.255274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1016.280545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1016.296337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1018.814523] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 16:38:28 (1779309508) [ 1018.853572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1018.869934] LNet: 114707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.883210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.919794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1019.072851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1019.091932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1019.125471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1019.313323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1019.329908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1019.360706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1019.379957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1019.413935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1019.565309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1019.581976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1019.614952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1019.780761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1019.796698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1022.194868] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 16:38:32 (1779309512) [ 1022.235079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1022.244433] LNet: 115341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.248076] LNet: 115341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1022.262925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1022.294860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1022.307104] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1022.603696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1022.614357] LNet: Added LNI 192.168.202.53@tcp2 [8/256/0/180] [ 1022.994108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1023.019474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1023.105179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1023.116225] LNetError: 115626:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.119584] LNet: Removed LNI 192.168.202.53@tcp [ 1023.130379] LNet: Removed LNI 192.168.202.53@tcp2 [ 1023.140258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.172765] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1023.176737] LNet: Added LNI 192.168.202.53@tcp2 [8/256/0/180] [ 1023.179857] LNet: 115632:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1023.369789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1023.453149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1024.479265] LNet: Removed LNI 192.168.202.53@tcp [ 1024.480598] LNetError: 115669:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.483393] LNetError: 115669:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1024.492384] LNet: Removed LNI 192.168.202.53@tcp2 [ 1024.506307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.531911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1024.543983] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1024.857833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1024.868076] LNet: Added LNI 192.168.202.53@tcp2 [8/256/0/180] [ 1025.206602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1025.230782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1025.258151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1025.335289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1025.343790] LNet: 115954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.346476] LNet: 115954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1026.399143] LNet: Removed LNI 192.168.202.53@tcp [ 1027.429542] LNet: Removed LNI 192.168.202.53@tcp2 [ 1027.439659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.467186] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1027.471419] LNet: Added LNI 192.168.202.53@tcp2 [8/256/0/180] [ 1027.473851] LNet: 115960:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1027.659735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1027.736084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1027.745228] LNetError: 115997:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.748323] LNetError: 115997:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1027.751862] LNet: Removed LNI 192.168.202.53@tcp [ 1028.773380] LNet: Removed LNI 192.168.202.53@tcp2 [ 1028.782288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1028.805110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1028.815610] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1029.119234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1029.129380] LNet: Added LNI 192.168.202.53@tcp2 [8/256/0/180] [ 1029.492819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1029.519878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1029.618733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1029.629863] LNet: 116276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.635109] LNet: 116276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1030.689673] LNet: Removed LNI 192.168.202.53@tcp [ 1030.697452] LNet: Removed LNI 192.168.202.53@tcp2 [ 1030.706942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.732187] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1030.735989] LNet: Added LNI 192.168.202.53@tcp2 [8/256/0/180] [ 1030.738377] LNet: 116283:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1030.959721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1031.077803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1032.095271] LNet: Removed LNI 192.168.202.53@tcp [ 1032.097155] LNetError: 116320:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1032.100149] LNetError: 116320:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1032.109454] LNet: Removed LNI 192.168.202.53@tcp2 [ 1032.119533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1032.146417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1032.157873] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1032.508707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1032.523086] LNet: Added LNI 192.168.202.53@tcp2 [8/256/0/180] [ 1032.929439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1032.952934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1032.977213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1033.066109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1034.079122] LNet: Removed LNI 192.168.202.53@tcp [ 1035.108449] LNet: Removed LNI 192.168.202.53@tcp2 [ 1035.119184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1035.150801] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1035.154826] LNet: Added LNI 192.168.202.53@tcp2 [8/256/0/180] [ 1035.157590] LNet: 116610:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1035.348974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1035.427105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1035.437780] LNet: Removed LNI 192.168.202.53@tcp [ 1036.453361] LNet: Removed LNI 192.168.202.53@tcp2 [ 1036.464303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.488768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1036.498546] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1036.810159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1036.821720] LNet: Added LNI 192.168.202.53@tcp2 [8/256/0/180] [ 1037.182675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1037.199501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1037.229988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1037.401881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1038.431350] LNet: Removed LNI 192.168.202.53@tcp [ 1038.440730] LNet: Removed LNI 192.168.202.53@tcp2 [ 1038.456578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1038.517686] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1038.520710] LNet: Added LNI 192.168.202.53@tcp2 [8/256/0/180] [ 1038.720566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1038.864080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1038.875245] LNet: 117002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.877621] LNet: 117002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1039.904323] LNet: Removed LNI 192.168.202.53@tcp [ 1039.912457] LNet: Removed LNI 192.168.202.53@tcp2 [ 1039.923588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1039.952725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1039.964537] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1040.295455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1040.305064] LNet: Added LNI 192.168.202.53@tcp2 [8/256/0/180] [ 1040.650790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1040.670119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1040.683983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1040.711676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1040.845346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1040.854296] LNetError: 117303:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1040.856667] LNetError: 117303:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1040.860710] LNet: Removed LNI 192.168.202.53@tcp [ 1041.894565] LNet: Removed LNI 192.168.202.53@tcp2 [ 1041.906798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.940203] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1041.943451] LNet: Added LNI 192.168.202.53@tcp2 [8/256/0/180] [ 1041.946719] LNet: 117309:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1041.950524] LNet: 117309:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1042.135523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1045.022330] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1045.720504] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 16:38:55 (1779309535) [ 1045.754964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1045.768425] LNet: Removed LNI 192.168.202.53@tcp [ 1046.822516] LNet: Removed LNI 192.168.202.53@tcp2 [ 1046.832775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1046.948073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1046.961607] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1047.319052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1047.329341] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1047.733714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.744820] LNet: Removed LNI 192.168.202.53@tcp [ 1047.753349] LNet: Removed LNI 10.1.2.3@tcp [ 1047.761695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.795738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1047.811706] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1047.814871] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1048.000854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1048.015124] LNet: Removed LNI 192.168.202.53@tcp [ 1048.021720] LNet: Removed LNI 10.1.2.3@tcp [ 1050.733780] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 16:39:00 (1779309540) [ 1050.776943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1050.796603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1050.837221] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1050.894935] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1050.897553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1050.900330] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1053.068129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1053.089654] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1053.450946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1053.462911] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1054.377344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.53@tcp [ 1054.397701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1054.425596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.53@tcp -m PUT -e local_timeout -r 1 [ 1054.458622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1054.485946] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1054.490169] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1054.587852] LNetError: 119005:0:(lib-move.c:4731:lnet_parse()) 192.168.202.53@tcp, src 192.168.202.53@tcp: Dropping PUT (error -108 looking up sender) [ 1055.647449] LNet: 119464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.650173] LNet: 119464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1056.671154] LNet: Removed LNI 192.168.202.53@tcp [ 1056.679857] LNet: Removed LNI 10.1.2.3@tcp [ 1056.838494] Key type .llcrypt unregistered [ 1056.841285] Key type ._llcrypt unregistered [ 1061.511066] Key type ._llcrypt registered [ 1061.512460] Key type .llcrypt registered [ 1062.143447] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 16:39:11 (1779309551) [ 1062.321035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1062.329180] alg: No test for adler32 (adler32-zlib) [ 1063.153984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1063.343374] LNet: 120315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.615278] Key type .llcrypt unregistered [ 1063.617757] Key type ._llcrypt unregistered [ 1065.985801] Key type ._llcrypt registered [ 1065.987667] Key type .llcrypt registered [ 1066.069997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1066.077461] alg: No test for adler32 (adler32-zlib) [ 1066.893851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1066.909675] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1070.249376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.153@tcp [ 1070.346740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1070.603174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.153@tcp [ 1070.706499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1070.721378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1070.740906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1070.760724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1070.778646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1070.798058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1070.816336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1070.832953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1070.846362] LNet: 120764:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1070.857554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1070.877250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1070.897719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1070.920629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1070.938814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1070.953334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1071.219792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1071.328258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1071.344671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1071.359732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1071.374839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1071.390903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1071.406480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1071.423281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1071.439990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1071.449550] LNet: 120881:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1071.459553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1071.476151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1071.497645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1071.516262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1071.533466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1071.556232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1071.835307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.153@tcp [ 1071.925211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1071.939947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1071.955801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1071.970220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1071.985108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1072.001221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1072.021149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1072.031291] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.153@tcp [ 1072.039832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1072.057139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1072.080580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1072.097526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1072.112452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1072.127280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1072.145313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1072.412992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1072.508603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1072.525607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1072.540840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1072.556598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1072.572415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1072.587118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1072.603731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1072.626160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1072.638861] LNet: 121115:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1072.642703] LNet: 121115:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1072.653358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1072.670254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1072.684604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1072.699993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1072.716273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1072.733379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1072.875330] LNet: 121161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.878637] LNetError: 121161:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1072.891373] LNet: Removed LNI 192.168.202.53@tcp [ 1073.064550] Key type .llcrypt unregistered [ 1073.066457] Key type ._llcrypt unregistered [ 1079.296324] Key type ._llcrypt registered [ 1079.297914] Key type .llcrypt registered [ 1079.911350] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 16:39:29 (1779309569) [ 1080.128604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1080.137687] alg: No test for adler32 (adler32-zlib) [ 1080.954621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.130101] LNet: 122049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.391519] Key type .llcrypt unregistered [ 1081.393736] Key type ._llcrypt unregistered [ 1083.586895] Key type ._llcrypt registered [ 1083.588286] Key type .llcrypt registered [ 1083.664061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1083.670493] alg: No test for adler32 (adler32-zlib) [ 1084.483868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1084.496362] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1087.768345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.153@tcp [ 1091.281715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1091.307584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1091.319977] LNet: Added LNI 192.168.202.53@tcp1 [8/256/0/180] [ 1091.743756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1092.002986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.153@tcp [ 1092.103184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1092.117725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1092.136536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1092.153584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1092.171311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1092.188380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1092.206121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1092.225233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1092.235454] LNet: 122699:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1092.251815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1092.269420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1092.285216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1092.299520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1092.314583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1092.331778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1092.602435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1092.705676] LNet: 2 peer NIs in recovery (showing 2): 192.168.202.153@tcp, 192.168.202.153@tcp1 [ 1092.710685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1092.727426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1092.743111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1092.763449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1092.781330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1092.799312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1092.814980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1092.829709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1092.838599] LNet: 122816:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1092.848586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1092.865943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1092.883722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1092.900448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1092.916927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1092.934479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1093.204790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.153@tcp [ 1093.308114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1093.324470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1093.338578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1093.355968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1093.371737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1093.386125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1093.398740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1093.421483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1093.439432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1093.462761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1093.480391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1093.495799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1093.519223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1093.543064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1093.853540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1093.983381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1093.999154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1094.015942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1094.030166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1094.044889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1094.061625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1094.076482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1094.090559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1094.102372] LNet: 123050:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1094.108759] LNet: 123050:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1094.123404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1094.143697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1094.168580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1094.198406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1094.216735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1094.233539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1096.702224] LNet: 123131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1096.709439] LNetError: 123131:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1096.713621] LNet: Removed LNI 192.168.202.53@tcp [ 1096.725638] LNet: Removed LNI 192.168.202.53@tcp1 [ 1096.939696] Key type .llcrypt unregistered [ 1096.941237] Key type ._llcrypt unregistered [ 1103.707148] Key type ._llcrypt registered [ 1103.708651] Key type .llcrypt registered [ 1104.320892] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 16:39:54 (1779309594) [ 1107.016529] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 16:39:56 (1779309596) [ 1109.876679] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 16:39:59 (1779309599) [ 1112.809289] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 16:40:02 (1779309602) [ 1115.653121] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 16:40:05 (1779309605) [ 1118.413855] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 16:40:08 (1779309608) [ 1121.154811] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 16:40:11 (1779309611) [ 1123.986898] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 16:40:13 (1779309613) [ 1126.811721] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 16:40:16 (1779309616) [ 1129.450809] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 16:40:19 (1779309619) [ 1131.984834] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 16:40:21 (1779309621) [ 1134.685431] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 16:40:24 (1779309624) [ 1137.171634] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 16:40:27 (1779309627) [ 1139.703672] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 16:40:29 (1779309629) [ 1142.635871] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 16:40:32 (1779309632) [ 1145.451649] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 16:40:35 (1779309635) [ 1148.423561] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 16:40:38 (1779309638) [ 1151.221655] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 16:40:41 (1779309641) [ 1151.357267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1151.363247] alg: No test for adler32 (adler32-zlib) [ 1152.182581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1152.238600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1152.250380] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1152.710497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1152.724774] LNet: 133827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.729444] LNetError: 133827:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.741146] LNet: Removed LNI 192.168.202.53@tcp [ 1152.755437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1152.795722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1152.815812] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1155.532363] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 16:40:45 (1779309645) [ 1158.347990] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 16:40:48 (1779309648) [ 1161.075579] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 16:40:50 (1779309650) [ 1163.801338] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 16:40:53 (1779309653) [ 1166.290556] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 16:40:56 (1779309656) [ 1168.846964] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 16:40:58 (1779309658) [ 1171.220254] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 16:41:01 (1779309661) [ 1173.754167] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 16:41:03 (1779309663) [ 1176.341350] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 16:41:06 (1779309666) [ 1178.927251] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 16:41:08 (1779309668) [ 1181.455562] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 16:41:11 (1779309671) [ 1183.904832] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 16:41:13 (1779309673) [ 1186.341699] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1042 sec ========== 16:41:16 (1779309676) [ 1186.434418] LNet: 141282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.437259] LNetError: 141282:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1186.444749] LNet: Removed LNI 192.168.202.53@tcp [ 1186.673604] Key type .llcrypt unregistered [ 1186.674929] Key type ._llcrypt unregistered [ 1190.762758] Key type ._llcrypt registered [ 1190.764213] Key type .llcrypt registered [ 1191.015147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1191.022960] alg: No test for adler32 (adler32-zlib) [ 1191.906869] Lustre: Lustre: Build Version: 2.17.53_24_ge4eab26 [ 1192.001734] LNet: Added LNI 192.168.202.53@tcp [8/256/0/180] [ 1193.591198] Key type lgssc registered [ 1194.085058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1217.571672] Lustre: Mounted lustre-client [ 1220.165994] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1222.586250] LustreError: 143720:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8c4ed810e000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1222.611043] Lustre: Unmounted lustre-client [ 1254.769979] Key type lgssc unregistered [ 1254.895890] LNet: 144201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1254.900199] LNetError: 144201:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1254.907506] LNet: Removed LNI 192.168.202.53@tcp [ 1255.150131] Key type .llcrypt unregistered [ 1255.151512] Key type ._llcrypt unregistered