[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 412348235 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002399] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007705] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009006] pid_max: default: 32768 minimum: 301 [ 0.010126] LSM: Security Framework initializing [ 0.012011] Yama: becoming mindful. [ 0.013026] SELinux: Initializing. [ 0.014049] *** VALIDATE selinux *** [ 0.022089] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027053] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028118] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029083] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030077] *** VALIDATE tmpfs *** [ 0.032345] *** VALIDATE proc *** [ 0.033196] *** VALIDATE cgroup *** [ 0.034005] *** VALIDATE cgroup2 *** [ 0.036024] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038107] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040027] Spectre V2 : User space: Vulnerable [ 0.041005] Speculative Store Bypass: Vulnerable [ 0.044146] debug: unmapping init [mem 0xffffffff94859000-0xffffffff94860fff] [ 0.046219] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047638] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048012] ... version: 2 [ 0.049007] ... bit width: 48 [ 0.050007] ... generic registers: 4 [ 0.051007] ... value mask: 0000ffffffffffff [ 0.052007] ... max period: 00007fffffffffff [ 0.053007] ... fixed-purpose events: 3 [ 0.054007] ... event mask: 000000070000000f [ 0.055394] rcu: Hierarchical SRCU implementation. [ 0.057850] smp: Bringing up secondary CPUs ... [ 0.058719] x86: Booting SMP configuration: [ 0.059013] .... node #0, CPUs: #1 #2 #3 [ 0.062774] smp: Brought up 1 node, 4 CPUs [ 0.064007] smpboot: Max logical packages: 1 [ 0.065013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.251615] node 0 deferred pages initialised in 184ms [ 0.254561] devtmpfs: initialized [ 0.255224] x86/mm: Memory block size: 128MB [ 0.257980] gcov: version magic: 0x41383552 [ 0.258553] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.259072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.260237] pinctrl core: initialized pinctrl subsystem [ 0.261097] [ 0.261474] ************************************************************* [ 0.262006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.263006] ** ** [ 0.264009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.265006] ** ** [ 0.266006] ** This means that this kernel is built to expose internal ** [ 0.267005] ** IOMMU data structures, which may compromise security on ** [ 0.268006] ** your system. ** [ 0.269006] ** ** [ 0.270005] ** If you see this message and you are not debugging the ** [ 0.271006] ** kernel, report this immediately to your vendor! ** [ 0.272008] ** ** [ 0.273008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.274006] ************************************************************* [ 0.275680] NET: Registered protocol family 16 [ 0.276552] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.277076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.278033] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.279414] cpuidle: using governor menu [ 0.281510] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.283475] PCI: Using configuration type 1 for base access [ 0.284094] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.292057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.293018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.295148] cryptd: max_cpu_qlen set to 1000 [ 0.297172] ACPI: Added _OSI(Module Device) [ 0.298009] ACPI: Added _OSI(Processor Device) [ 0.298908] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.300007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.303084] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.306387] ACPI: Interpreter enabled [ 0.308042] ACPI: PM: (supports S0 S3 S4 S5) [ 0.308894] ACPI: Using IOAPIC for interrupt routing [ 0.309055] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.311254] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.317975] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.319018] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.321008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.323046] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.326083] acpiphp: Slot [2] registered [ 0.326897] acpiphp: Slot [5] registered [ 0.328081] acpiphp: Slot [6] registered [ 0.329017] acpiphp: Slot [3] registered [ 0.329864] acpiphp: Slot [4] registered [ 0.330043] acpiphp: Slot [7] registered [ 0.330928] acpiphp: Slot [8] registered [ 0.332044] acpiphp: Slot [9] registered [ 0.332889] acpiphp: Slot [10] registered [ 0.334058] acpiphp: Slot [11] registered [ 0.334899] acpiphp: Slot [12] registered [ 0.335043] acpiphp: Slot [13] registered [ 0.335926] acpiphp: Slot [14] registered [ 0.337059] acpiphp: Slot [15] registered [ 0.337940] acpiphp: Slot [16] registered [ 0.339045] acpiphp: Slot [17] registered [ 0.339859] acpiphp: Slot [18] registered [ 0.340045] acpiphp: Slot [19] registered [ 0.340902] acpiphp: Slot [20] registered [ 0.342057] acpiphp: Slot [21] registered [ 0.342927] acpiphp: Slot [22] registered [ 0.344045] acpiphp: Slot [23] registered [ 0.344924] acpiphp: Slot [24] registered [ 0.345063] acpiphp: Slot [25] registered [ 0.345823] acpiphp: Slot [26] registered [ 0.347051] acpiphp: Slot [27] registered [ 0.348029] acpiphp: Slot [28] registered [ 0.348904] acpiphp: Slot [29] registered [ 0.350045] acpiphp: Slot [30] registered [ 0.350911] acpiphp: Slot [31] registered [ 0.351041] PCI host bridge to bus 0000:00 [ 0.351823] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.354015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.355010] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.357009] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.358029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.361044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.362236] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.365077] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.368151] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.375846] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.378618] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.381012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.383009] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.385011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.387633] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.389732] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.392048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.394814] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.398012] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.407016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.410014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416459] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.428021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.440014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.447184] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.453022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.458025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.473018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.483220] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.485319] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.487291] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.489272] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.491146] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.496420] iommu: Default domain type: Passthrough [ 0.498406] SCSI subsystem initialized [ 0.500100] ACPI: bus type USB registered [ 0.501075] usbcore: registered new interface driver usbfs [ 0.503039] usbcore: registered new interface driver hub [ 0.504061] usbcore: registered new device driver usb [ 0.506139] pps_core: LinuxPPS API ver. 1 registered [ 0.508009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.511024] PTP clock support registered [ 0.512097] EDAC MC: Ver: 3.0.0 [ 0.514103] PCI: Using ACPI for IRQ routing [ 0.515690] NetLabel: Initializing [ 0.517008] NetLabel: domain hash size = 128 [ 0.519021] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.521063] NetLabel: unlabeled traffic allowed by default [ 0.523088] vgaarb: loaded [ 0.524255] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.526009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.535374] clocksource: Switched to clocksource kvm-clock [ 0.620630] VFS: Disk quotas dquot_6.6.0 [ 0.621544] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.623156] *** VALIDATE ramfs *** [ 0.623966] *** VALIDATE hugetlbfs *** [ 0.624919] pnp: PnP ACPI init [ 0.626594] pnp: PnP ACPI: found 6 devices [ 0.641075] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.643096] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.644346] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.645854] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.647238] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.648722] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.650393] NET: Registered protocol family 2 [ 0.652080] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.655513] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.657660] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.661259] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.663204] TCP: Hash tables configured (established 65536 bind 65536) [ 0.664821] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.666522] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.668343] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.670116] NET: Registered protocol family 1 [ 0.671695] RPC: Registered named UNIX socket transport module. [ 0.672844] RPC: Registered udp transport module. [ 0.673703] RPC: Registered tcp transport module. [ 0.674676] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.675920] NET: Registered protocol family 44 [ 0.676830] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.677956] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.679125] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.680310] PCI: CLS 0 bytes, default 64 [ 0.681355] Unpacking initramfs... [ 2.050023] debug: unmapping init [mem 0xffffa01cbcc64000-0xffffa01cbffcffff] [ 2.052803] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.054304] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.056341] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.573335] Initialise system trusted keyrings [ 2.574865] Key type blacklist registered [ 2.576496] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.585327] zbud: loaded [ 2.588656] *** VALIDATE nfs *** [ 2.589789] *** VALIDATE nfs4 *** [ 2.590973] pstore: using deflate compression [ 2.593808] Platform Keyring initialized [ 2.660837] NET: Registered protocol family 38 [ 2.662532] Key type asymmetric registered [ 2.663532] Asymmetric key parser 'x509' registered [ 2.664573] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.666750] io scheduler mq-deadline registered [ 2.667870] io scheduler kyber registered [ 2.669104] io scheduler bfq registered [ 2.670174] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.672352] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.674686] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.676736] ACPI: Power Button [PWRF] [ 2.680459] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.687386] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.694648] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.719578] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.747973] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.753377] Non-volatile memory driver v1.3 [ 2.755100] Linux agpgart interface v0.103 [ 2.788413] virtio_blk virtio1: [vda] 134152 512-byte logical blocks (68.7 MB/65.5 MiB) [ 2.790960] vda: detected capacity change from 0 to 68685824 [ 2.802834] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.805104] vdb: detected capacity change from 0 to 1073741824 [ 2.810283] libphy: Fixed MDIO Bus: probed [ 2.814280] usbcore: registered new interface driver usbserial_generic [ 2.816814] usbserial: USB Serial support registered for generic [ 2.819202] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.823470] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.825156] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.827304] mousedev: PS/2 mouse device common for all mice [ 2.829815] rtc_cmos 00:05: RTC can wake from S4 [ 2.832407] rtc_cmos 00:05: registered as rtc0 [ 2.832827] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.834155] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.839343] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.840068] intel_pstate: CPU model not supported [ 2.844523] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.849203] hid: raw HID events driver (C) Jiri Kosina [ 2.851201] usbcore: registered new interface driver usbhid [ 2.853077] usbhid: USB HID core driver [ 2.854437] drop_monitor: Initializing network drop monitor service [ 2.856438] Initializing XFRM netlink socket [ 2.858911] NET: Registered protocol family 10 [ 2.862802] Segment Routing with IPv6 [ 2.863817] NET: Registered protocol family 17 [ 2.865074] mpls_gso: MPLS GSO support [ 2.869508] RAS: Correctable Errors collector initialized. [ 2.871279] AVX version of gcm_enc/dec engaged. [ 2.872408] AES CTR mode by8 optimization enabled [ 2.928313] sched_clock: Marking stable (2928296364, 0)->(3637084563, -708788199) [ 2.930340] registered taskstats version 1 [ 2.931645] Loading compiled-in X.509 certificates [ 2.933437] zswap: loaded using pool lzo/zbud [ 2.952586] Key type big_key registered [ 2.965501] Key type encrypted registered [ 2.967445] ima: No TPM chip found, activating TPM-bypass! [ 2.969937] ima: Allocated hash algorithm: sha1 [ 2.971449] ima: No architecture policies found [ 2.972640] evm: Initialising EVM extended attributes: [ 2.973983] evm: security.selinux [ 2.974871] evm: security.ima [ 2.975701] evm: security.capability [ 2.976678] evm: HMAC attrs: 0x1 [ 2.978430] rtc_cmos 00:05: setting system clock to 2026-01-11 13:59:44 UTC (1768139984) [ 2.983736] debug: unmapping init [mem 0xffffffff95803000-0xffffffff959fffff] [ 2.986232] debug: unmapping init [mem 0xffffffff94582000-0xffffffff94858fff] [ 2.994107] Write protecting the kernel read-only data: 28672k [ 2.996872] debug: unmapping init [mem 0xffffffff92c03000-0xffffffff92dfffff] [ 2.999071] debug: unmapping init [mem 0xffffffff93514000-0xffffffff935fffff] [ 3.026577] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.033304] systemd[1]: Detected virtualization kvm. [ 3.036800] systemd[1]: Detected architecture x86-64. [ 3.038682] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.060608] systemd[1]: No hostname configured. [ 3.061815] systemd[1]: Set hostname to . [ 3.063287] random: systemd: uninitialized urandom read (16 bytes read) [ 3.065304] systemd[1]: Initializing machine ID from random generator. [ 3.103172] random: ln: uninitialized urandom read (6 bytes read) [ 3.206650] random: systemd: uninitialized urandom read (16 bytes read) [ 3.210778] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.215285] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.218858] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.809285] device-mapper: uevent: version 1.0.3 [ 3.811732] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted Kernel Configuration File System. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.501050] virtio_net virtio0 ens2: renamed from eth0 [ 4.600136] scsi host0: ata_piix [ 4.656221] scsi host1: ata_piix [ 4.657911] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.660600] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 7.093389] random: fast init done [ 8.832637] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.481060] random: crng init done [ 9.482694] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.024501] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.120466] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.404349] SELinux: Disabled at runtime. [ 11.471662] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.482174] systemd[1]: Detected virtualization kvm. [ 11.484248] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.956884] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.959702] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.964091] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.968237] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.971382] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.977678] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.993588] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ 12.123530] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.354983] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.639399] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.664714] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.864827] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.878386] EDAC sbridge: Ver: 1.1.2 [ 14.341194] Key type dns_resolver registered [ 14.689161] NFS: Registering the id_resolver key type [ 14.691275] Key type id_resolver registered [ 14.692854] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg203-client login: [ 40.486385] libcfs: loading out-of-tree module taints kernel. [ 40.510483] Key type ._llcrypt registered [ 40.512750] Key type .llcrypt registered [ 40.781171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 40.785526] alg: No test for adler32 (adler32-zlib) [ 41.744409] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 41.998361] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 43.599114] Key type lgssc registered [ 44.074795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 86.849377] Lustre: Mounted lustre-client [ 89.248439] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 99.770958] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing check_logdir /tmp/testlogs/ [ 101.212969] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing yml_node [ 102.615425] Lustre: DEBUG MARKER: Client: 2.17.50.31 [ 103.411537] Lustre: DEBUG MARKER: MDS: 2.17.50.31 [ 104.228270] Lustre: DEBUG MARKER: OSS: 2.17.50.31 [ 104.741704] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Jan 11 09:01:25 EST 2026 [ 107.487259] Lustre: lustre-OST0000-osc-ffffa01d07c87000: disconnect after 20s idle [ 109.911901] LustreError: 7897:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffffa01d07c87000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 109.915444] LustreError: 7897:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 109.924088] Lustre: Unmounted lustre-client [ 118.232047] Key type lgssc unregistered [ 118.356514] LNet: 8343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 118.360121] LNetError: 8343:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 118.369704] LNet: Removed LNI 192.168.202.3@tcp [ 118.619112] Key type .llcrypt unregistered [ 118.620037] Key type ._llcrypt unregistered [ 127.170157] Key type ._llcrypt registered [ 127.171033] Key type .llcrypt registered [ 127.227745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 127.233983] alg: No test for adler32 (adler32-zlib) [ 128.034425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 128.062210] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 128.070573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 129.297613] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 129.329300] LNet: 9221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 129.332943] LNetError: 9221:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 129.339382] LNet: Removed LNI 192.168.202.3@tcp [ 129.544335] Key type .llcrypt unregistered [ 129.545370] Key type ._llcrypt unregistered [ 131.483989] Key type ._llcrypt registered [ 131.484940] Key type .llcrypt registered [ 131.892179] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:01:52 (1768140112) [ 132.154670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 132.160638] alg: No test for adler32 (adler32-zlib) [ 132.971415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 133.020412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 133.032788] LNet: 9632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 133.044820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 133.075083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 133.088178] LNet: 9639:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 135.552074] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:01:56 (1768140116) [ 135.638772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 135.644952] LNet: 10216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 135.654160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 135.684393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 138.105303] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:01:59 (1768140119) [ 138.141434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 138.150503] LNet: 10797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 138.160565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 138.192400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 140.316424] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:02:01 (1768140121) [ 140.380807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 140.388330] LNet: 11383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 140.396336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 140.423544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 142.420641] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:02:03 (1768140123) [ 142.532262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 142.545440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 142.570430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 142.699177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 145.092842] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 09:02:06 (1768140126) [ 145.124436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 145.134765] LNet: 12551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 145.136789] LNet: 12551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 145.147594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 145.180678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 145.320203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 145.461736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 147.586108] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 09:02:08 (1768140128) [ 147.630270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 147.650199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 147.679767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 150.113028] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 09:02:11 (1768140131) [ 150.151712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 150.169943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 150.202896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 150.218647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 150.344580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 150.360300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 150.486946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 150.498481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 150.611827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 150.625511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 150.756899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 150.770725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 150.911935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 150.925339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 151.057592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 151.069671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 151.204592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 151.217045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 151.340968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 151.353317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 151.470980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 151.483245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 151.641245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 151.656166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 151.794248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 151.809718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 151.949139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 151.962417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 154.101831] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 09:02:15 (1768140135) [ 154.160273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 154.173769] LNet: 14415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 154.176050] LNet: 14415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 154.185090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 154.208944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 154.221301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 156.413289] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 09:02:17 (1768140137) [ 156.462645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 156.478246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 156.503938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 156.516517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 158.560923] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 09:02:19 (1768140139) [ 158.614903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 158.629724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 158.652684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 158.663341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 160.739474] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 09:02:21 (1768140141) [ 160.774807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 160.791931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 160.828548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 160.842838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 162.926187] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 09:02:24 (1768140144) [ 162.996023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 163.009813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 163.035912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 163.051099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 165.116647] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 09:02:26 (1768140146) [ 165.165777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 165.183843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 165.213000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 165.229942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 167.492894] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 09:02:28 (1768140148) [ 167.531089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 167.549695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 167.593448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 167.722951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 170.239866] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 09:02:31 (1768140151) [ 170.273467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 170.281974] LNet: 18517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 170.284440] LNet: 18517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 170.296121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 170.373382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 170.457760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 172.856049] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 09:02:33 (1768140153) [ 172.916555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 172.951390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 173.009331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 173.112137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 175.971403] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 09:02:36 (1768140156) [ 176.000341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 176.017586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 176.195021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 176.300620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 178.348911] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 09:02:39 (1768140159) [ 178.383691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 178.398760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 178.423715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 178.439350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 180.571743] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 09:02:41 (1768140161) [ 180.616676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 180.638790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 180.666987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 180.683676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 182.871018] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 09:02:43 (1768140163) [ 182.899455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 182.915886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 182.947669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 185.029099] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 09:02:46 (1768140166) [ 185.070095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 185.087372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 185.109511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 187.107019] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 09:02:48 (1768140168) [ 187.139881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 187.156885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 187.186917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 187.322712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 189.480791] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 09:02:50 (1768140170) [ 189.514583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 189.536502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 189.566799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 189.577922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 191.587801] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 09:02:52 (1768140172) [ 191.615600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 191.631708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 191.659556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 191.673728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 193.640852] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 09:02:54 (1768140174) [ 193.699808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 193.713019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 193.734475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 193.746913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 195.834795] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 09:02:56 (1768140176) [ 195.875726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 195.896110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 195.927871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 195.941443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 198.312936] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 09:02:59 (1768140179) [ 198.343645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 198.360810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 198.385891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 198.413477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 198.424653] LNetError: 25716:0:(peer.c:2167:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 198.447338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 198.462046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 200.572142] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 09:03:01 (1768140181) [ 200.618382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 200.637086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 200.665432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.qywfCAFqz3 [ 200.695740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.qywfCAFqz3 [ 200.716774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.qywfCAFqz3 [ 202.730796] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 09:03:03 (1768140183) [ 202.801841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 202.810549] LNet: 26886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 202.813582] LNet: 26886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 202.823471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 202.845542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 202.859277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 202.877564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 205.095304] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 09:03:06 (1768140186) [ 205.128661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 205.144365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 205.171311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 206.118134] LNet: Message forwarding will be enabled in 2s [ 209.183084] LNet: Message forwarding enabled [ 209.256780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 209.267399] LNet: Message forwarding will stop in 2s to 6s [ 213.426446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 213.971516] LNet: Message forwarding disabled [ 214.400105] LNet: Message forwarding will be enabled in 2s [ 216.133031] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 09:03:17 (1768140197) [ 216.200489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 216.707905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 216.728840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 218.607524] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 09:03:19 (1768140199) [ 218.647226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 218.665185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 218.684930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 218.696636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 218.707257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 218.718033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 218.734553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 218.746827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 218.757160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 218.768720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 218.782339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 218.794171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 218.804435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 218.818362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 218.831458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 218.843354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 218.854748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 218.865647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 218.878197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 218.890196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 218.901052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 218.908926] LNetError: 28782:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 220.780719] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 09:03:21 (1768140201) [ 220.818315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.834885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.859817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 222.874149] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 09:03:23 (1768140203) [ 223.443067] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg203-server [ 224.004429] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 09:03:25 (1768140205) [ 224.029868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 224.046589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 224.073528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 224.082317] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 224.605329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[4-8]@tcp [ 224.614027] LNet: 30496:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 226.616275] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 09:03:27 (1768140207) [ 227.168575] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg203-server [ 227.671421] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 09:03:28 (1768140208) [ 227.679478] LNetError: 30497:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.7@tcp has gone from up to down [ 227.709782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 228.703145] LNetError: 30285:0:(router.c:1267:lnet_check_routers()) Failed to discover router 192.168.202.4@tcp [ 230.815140] LNetError: 30286:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.8@tcp has gone from up to down [ 230.817805] LNetError: 30286:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 230.820407] LNetError: 31426:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 230.829687] LNet: Removed LNI 192.168.202.3@tcp [ 230.836520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.857217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 230.865803] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 231.093054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[4-9/2]@tcp [ 231.100431] LNet: 31547:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 231.103101] LNet: 31547:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 231.110457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[4-9/2]@tcp [ 233.067174] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 09:03:34 (1768140214) [ 233.095971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.167591] LNetError: 32133:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 235.177398] LNet: Removed LNI 192.168.202.3@tcp [ 235.185281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.213437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 235.233392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.249726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.271817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 235.290628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.302603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.318117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 235.343595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.356882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.377840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 235.404360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.416573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.432406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 235.454274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.465996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.485918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 235.510489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.521681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.538156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 235.561609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.572683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.590921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 235.615518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.628784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.647767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 235.673702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.686447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.709282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 235.734109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.746160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.768540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 235.786584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.796816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.814476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 237.770450] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 09:03:38 (1768140218) [ 238.387779] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg203-server [ 238.922503] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 09:03:39 (1768140219) [ 239.491725] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg203-server [ 240.041709] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 09:03:41 (1768140221) [ 240.326872] Key type .llcrypt unregistered [ 240.327797] Key type ._llcrypt unregistered [ 242.085765] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 242.131979] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 242.134327] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 242.136607] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 244.317557] Key type ._llcrypt registered [ 244.318631] Key type .llcrypt registered [ 244.374380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 244.379637] alg: No test for adler32 (adler32-zlib) [ 245.181475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 245.200360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 245.209113] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 245.429578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 245.437595] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 246.060985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 246.068698] LNet: Removed LNI 10.1.2.3@tcp [ 246.109994] LNet: 34250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 246.111970] LNetError: 34250:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 246.120335] LNet: Removed LNI 192.168.202.3@tcp [ 246.321355] Key type .llcrypt unregistered [ 246.322333] Key type ._llcrypt unregistered [ 249.667644] Key type ._llcrypt registered [ 249.668667] Key type .llcrypt registered [ 250.080130] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 09:03:51 (1768140231) [ 250.234183] Key type .llcrypt unregistered [ 250.235401] Key type ._llcrypt unregistered [ 252.047244] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 252.097850] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 252.103268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 252.107075] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 254.294733] Key type ._llcrypt registered [ 254.296286] Key type .llcrypt registered [ 254.357505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 254.362945] alg: No test for adler32 (adler32-zlib) [ 255.162849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 255.180511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 255.188074] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 255.411478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 255.418887] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 256.220161] LNet: 35732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 256.222698] LNetError: 35732:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 256.226147] LNet: Removed LNI 192.168.202.3@tcp [ 256.239295] LNet: Removed LNI 10.1.2.3@tcp [ 256.437317] Key type .llcrypt unregistered [ 256.438322] Key type ._llcrypt unregistered [ 259.768752] Key type ._llcrypt registered [ 259.769770] Key type .llcrypt registered [ 260.174390] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 09:04:01 (1768140241) [ 260.335332] Key type .llcrypt unregistered [ 260.336370] Key type ._llcrypt unregistered [ 262.079800] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 262.158045] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 262.160328] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 262.162530] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 264.406515] Key type ._llcrypt registered [ 264.407529] Key type .llcrypt registered [ 264.472613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 264.480774] alg: No test for adler32 (adler32-zlib) [ 265.289262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.314345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 265.326479] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 265.969894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 265.977841] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 266.648178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 266.655504] LNet: Removed LNI 10.1.2.3@tcp [ 266.664208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 266.681339] LNet: Removed LNI 10.1.2.31@tcp [ 266.688873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 266.698135] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 266.962328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 266.969749] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 267.299512] LNet: 37643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 267.301780] LNetError: 37643:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 267.304041] LNet: Removed LNI 10.1.2.3@tcp [ 267.312356] LNet: Removed LNI 10.1.2.31@tcp [ 267.539778] Key type .llcrypt unregistered [ 267.541410] Key type ._llcrypt unregistered [ 271.191042] Key type ._llcrypt registered [ 271.192115] Key type .llcrypt registered [ 271.623360] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:04:12 (1768140252) [ 271.753206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 271.758523] alg: No test for adler32 (adler32-zlib) [ 272.565197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.593686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 272.605050] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 272.879367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 272.887198] LNet: 38590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 272.889981] LNetError: 38590:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 272.909310] LNet: Removed LNI 192.168.202.3@tcp [ 272.917261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.936451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.3@tcp [ 272.947040] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 273.233446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.3@tcp [ 273.252666] LNet: Removed LNI 192.168.202.3@tcp [ 273.536487] Key type .llcrypt unregistered [ 273.538364] Key type ._llcrypt unregistered [ 277.147837] Key type ._llcrypt registered [ 277.148816] Key type .llcrypt registered [ 277.567102] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:04:18 (1768140258) [ 278.156981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg203-server [ 278.678439] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:04:19 (1768140259) [ 278.820155] Key type .llcrypt unregistered [ 278.821190] Key type ._llcrypt unregistered [ 280.675199] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 280.768670] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 280.771308] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 280.774093] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 282.943960] Key type ._llcrypt registered [ 282.945250] Key type .llcrypt registered [ 283.011820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 283.017783] alg: No test for adler32 (adler32-zlib) [ 283.816412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 283.837604] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 284.066025] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 284.737386] LNet: 40549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 284.739425] LNetError: 40549:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 284.741595] LNet: Removed LNI 192.168.202.3@tcp1 [ 284.753292] LNet: Removed LNI 10.1.2.3@tcp2 [ 284.974265] Key type .llcrypt unregistered [ 284.975275] Key type ._llcrypt unregistered [ 288.435786] Key type ._llcrypt registered [ 288.438181] Key type .llcrypt registered [ 288.854088] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:04:29 (1768140269) [ 288.987888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 288.993692] alg: No test for adler32 (adler32-zlib) [ 289.798166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.820908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 289.829643] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 290.110281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.118782] LNet: 41511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.120717] LNetError: 41511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 290.128272] LNet: Removed LNI 192.168.202.3@tcp [ 290.135858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.157995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 290.165731] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 292.266820] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:04:33 (1768140273) [ 292.305934] LNet: 42096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 292.309328] LNetError: 42096:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 292.320507] LNet: Removed LNI 192.168.202.3@tcp [ 292.530307] Key type .llcrypt unregistered [ 292.531416] Key type ._llcrypt unregistered [ 294.434426] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 294.466860] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 294.469805] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 294.473921] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 296.684680] Key type ._llcrypt registered [ 296.685687] Key type .llcrypt registered [ 296.746262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 296.753588] alg: No test for adler32 (adler32-zlib) [ 297.564906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.583192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 297.590760] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 297.832673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 297.840605] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 298.555549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 298.561969] LNet: 42841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 298.563818] LNetError: 42841:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 298.565799] LNet: Removed LNI 192.168.202.3@tcp [ 298.578273] LNet: Removed LNI 10.1.2.3@tcp [ 298.586066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.606452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 298.615256] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 298.616796] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 298.761350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 298.771285] LNet: Removed LNI 192.168.202.3@tcp [ 298.778423] LNet: Removed LNI 10.1.2.3@tcp [ 298.786591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.810917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 298.819176] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 298.820618] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 298.961770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 298.968284] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 299.292826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.299743] LNet: 43030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.301789] LNet: 43030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 299.304493] LNetError: 43030:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 299.306799] LNetError: 43030:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 299.309197] LNet: Removed LNI 192.168.202.3@tcp [ 299.310510] LNet: Removed LNI 10.1.2.3@tcp [ 299.315467] LNet: Removed LNI 192.168.202.3@tcp2 [ 299.322620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.342144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 299.351501] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 299.354965] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 299.358140] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 301.514492] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 09:04:42 (1768140282) [ 301.557934] LNet: 43621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.560542] LNetError: 43621:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 301.562817] LNetError: 43621:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 301.565394] LNet: Removed LNI 192.168.202.3@tcp [ 301.566625] LNet: Removed LNI 10.1.2.3@tcp [ 301.573561] LNet: Removed LNI 192.168.202.3@tcp2 [ 301.804601] Key type .llcrypt unregistered [ 301.805714] Key type ._llcrypt unregistered [ 303.847346] Key type ._llcrypt registered [ 303.849118] Key type .llcrypt registered [ 303.936479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 303.944842] alg: No test for adler32 (adler32-zlib) [ 304.751837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.771896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 304.781594] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 305.064181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.071474] LNet: 44016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 305.073729] LNetError: 44016:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 305.083339] LNet: Removed LNI 192.168.202.3@tcp [ 305.090532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 305.111619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 305.120555] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 305.269303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.281471] LNet: Removed LNI 192.168.202.3@tcp [ 305.288818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 305.309243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 305.318421] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 305.467605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.486438] LNet: Removed LNI 192.168.202.3@tcp [ 305.494641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 305.514624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 305.523813] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 305.670521] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 305.710817] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 305.714686] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 305.716194] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 307.875135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.886893] LNet: 44151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.890116] LNet: 44151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 307.892851] LNetError: 44151:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 307.895537] LNetError: 44151:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 307.910828] LNet: Removed LNI 192.168.202.3@tcp [ 307.922971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.955809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 307.969695] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 310.057079] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 09:04:51 (1768140291) [ 310.088700] LNet: 44740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.091834] LNetError: 44740:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 310.101748] LNet: Removed LNI 192.168.202.3@tcp [ 310.324708] Key type .llcrypt unregistered [ 310.325981] Key type ._llcrypt unregistered [ 312.009588] Key type ._llcrypt registered [ 312.010789] Key type .llcrypt registered [ 312.069108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 312.074461] alg: No test for adler32 (adler32-zlib) [ 312.870437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 312.890867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 312.901568] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 313.212665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.223942] LNet: 45141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.226738] LNetError: 45141:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 313.243420] LNet: Removed LNI 192.168.202.3@tcp [ 313.250941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.274871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 313.284929] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 315.600122] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 09:04:56 (1768140296) [ 315.630518] LNet: 45726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 315.632663] LNetError: 45726:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 315.641472] LNet: Removed LNI 192.168.202.3@tcp [ 315.844560] Key type .llcrypt unregistered [ 315.845630] Key type ._llcrypt unregistered [ 317.642473] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 317.716573] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 317.718056] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 317.719693] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 319.883265] Key type ._llcrypt registered [ 319.884140] Key type .llcrypt registered [ 319.946383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 319.952213] alg: No test for adler32 (adler32-zlib) [ 320.754938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.775623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 320.783627] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 321.072812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 321.080960] LNet: 46178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.082875] LNetError: 46178:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 321.090285] LNet: Removed LNI 192.168.202.3@tcp [ 321.097741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.117663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 321.127343] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 323.278412] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 09:05:04 (1768140304) [ 323.310848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.320090] LNet: 46769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 323.323736] LNetError: 46769:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 323.333043] LNet: Removed LNI 192.168.202.3@tcp [ 323.344023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.384783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 325.451315] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 09:05:06 (1768140306) [ 325.480550] LNet: 47350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.704610] Key type .llcrypt unregistered [ 325.705623] Key type ._llcrypt unregistered [ 327.593084] Key type ._llcrypt registered [ 327.594322] Key type .llcrypt registered [ 327.656560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 327.662310] alg: No test for adler32 (adler32-zlib) [ 328.465896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.486415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 328.496307] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 328.801715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.811858] LNet: 47749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.815389] LNetError: 47749:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 328.824420] LNet: Removed LNI 192.168.202.3@tcp [ 328.833626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.858146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 328.868304] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 330.975959] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 09:05:12 (1768140312) [ 331.017820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.025396] LNet: 48341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.029745] LNetError: 48341:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 331.040374] LNet: Removed LNI 192.168.202.3@tcp [ 331.047579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.068023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 332.980936] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 09:05:14 (1768140314) [ 333.462650] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 333.997220] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 09:05:15 (1768140315) [ 334.496361] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 335.011608] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 09:05:16 (1768140316) [ 335.513031] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 336.057372] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 09:05:17 (1768140317) [ 336.584581] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 337.087641] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 09:05:18 (1768140318) [ 337.559911] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 338.071788] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:05:19 (1768140319) [ 338.110361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.119318] LNet: 50391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.131358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.152501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 338.161506] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 338.409114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.3 [ 338.416451] LNet: 50528:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 338.420864] LNet: Removed LNI 192.168.202.3@tcp [ 340.393936] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:05:21 (1768140321) [ 340.430434] LNet: 51092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.432485] LNetError: 51092:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 340.442499] LNet: Removed LNI 192.168.202.3@tcp [ 340.634689] Key type .llcrypt unregistered [ 340.636442] Key type ._llcrypt unregistered [ 342.618689] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 342.711597] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 342.712993] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 342.714522] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 344.970144] Key type ._llcrypt registered [ 344.971183] Key type .llcrypt registered [ 345.022073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 345.027245] alg: No test for adler32 (adler32-zlib) [ 345.962431] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 347.972736] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:05:29 (1768140329) [ 348.001558] LNet: 51990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.004544] LNetError: 51990:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 348.012540] LNet: Removed LNI 10.1.2.3@tcp [ 348.225328] Key type .llcrypt unregistered [ 348.226433] Key type ._llcrypt unregistered [ 350.049883] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 350.152391] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 350.153856] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 350.155522] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 352.408274] Key type ._llcrypt registered [ 352.409729] Key type .llcrypt registered [ 352.463341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 352.468982] alg: No test for adler32 (adler32-zlib) [ 353.295992] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 355.416599] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:05:36 (1768140336) [ 355.458825] LNet: 52881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.461296] LNetError: 52881:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 355.470752] LNet: Removed LNI 10.1.2.3@tcp [ 355.695320] Key type .llcrypt unregistered [ 355.696561] Key type ._llcrypt unregistered [ 357.469378] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 357.602872] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 357.604417] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 357.606575] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 359.888492] Key type ._llcrypt registered [ 359.889646] Key type .llcrypt registered [ 359.947390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 359.954385] alg: No test for adler32 (adler32-zlib) [ 360.771945] LNet: 53181:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 360.776376] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 362.833957] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:05:43 (1768140343) [ 362.868719] LNet: 53766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.872363] LNetError: 53766:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 362.881608] LNet: Removed LNI 10.1.2.3@tcp [ 363.082414] Key type .llcrypt unregistered [ 363.083519] Key type ._llcrypt unregistered [ 364.973514] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 365.065759] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 365.067089] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 365.068401] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 367.337723] Key type ._llcrypt registered [ 367.339115] Key type .llcrypt registered [ 367.395529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 367.400400] alg: No test for adler32 (adler32-zlib) [ 368.200507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.227477] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 370.860178] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:05:51 (1768140351) [ 370.992596] LNet: 54996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.995036] LNetError: 54996:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 371.002477] LNet: Removed LNI 10.1.2.3@tcp [ 371.208622] Key type .llcrypt unregistered [ 371.209606] Key type ._llcrypt unregistered [ 373.097715] Key type ._llcrypt registered [ 373.098577] Key type .llcrypt registered [ 373.154370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 373.158752] alg: No test for adler32 (adler32-zlib) [ 373.963526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 373.973544] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 376.804733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 377.009624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_interrupt -r 1 [ 377.023819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_interrupt -r 1 [ 377.038162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 377.045082] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 377.055062] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 377.116550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 377.126975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 377.136673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 377.146268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 377.155918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 377.165930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 377.176403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 377.185993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 377.193244] LNet: 55428:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 377.198962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 377.208431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 377.219022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 377.228980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 377.238659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 377.248673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 377.258431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 377.413892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_dropped -r 1 [ 377.431322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_dropped -r 1 [ 377.443939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 377.518848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 377.527859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 377.539127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 377.549629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 377.559816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 377.570062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 377.580816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 377.591038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 377.601089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 377.611643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 377.621401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 377.630371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 377.640230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 377.653622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 377.667770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 377.847235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_aborted -r 1 [ 377.858157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_aborted -r 1 [ 377.869175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 377.875586] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 377.877403] LNet: Skipped 1 previous similar message [ 377.947390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 377.957934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 377.968796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 377.979812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 377.992046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 378.003403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 378.014453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 378.025206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 378.031364] LNet: 55646:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 378.033418] LNet: 55646:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 378.039302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 378.049181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 378.059842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 378.069318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 378.079295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 378.089866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 378.099838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 378.269465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_no_route -r 1 [ 378.283205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_no_route -r 1 [ 378.295480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 378.369984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 378.381119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 378.391396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 378.401462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 378.411614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 378.421114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 378.432220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 378.442625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 378.454287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 378.465227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 378.476777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 378.488158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 378.499542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 378.510843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 378.521623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 378.682584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_timeout -r 1 [ 378.695261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_timeout -r 1 [ 378.706445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 378.781961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 378.792227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 378.802729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 378.811755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 378.822373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 378.833130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 378.843995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 378.854984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 378.866094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 378.877329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 378.888100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 378.899355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 378.909893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 378.920964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 378.931729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 379.092180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_error -r 1 [ 379.102949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_error -r 1 [ 379.113538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 379.119600] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 379.121589] LNet: Skipped 3 previous similar messages [ 379.188784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 379.198125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 379.207912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 379.218203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 379.229545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 379.240604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 379.251815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 379.263922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 379.270504] LNet: 55973:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 379.272773] LNet: 55973:0:(api-ni.c:343:recovery_interval_set()) Skipped 2 previous similar messages [ 379.279280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 379.289227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 379.299649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 379.310294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 379.321316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 379.332073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 379.343539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 379.429126] LNet: 56013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.431321] LNetError: 56013:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 379.441342] LNet: Removed LNI 192.168.202.3@tcp [ 379.572377] Key type .llcrypt unregistered [ 379.573470] Key type ._llcrypt unregistered [ 384.693514] Key type ._llcrypt registered [ 384.694605] Key type .llcrypt registered [ 385.130979] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:06:06 (1768140366) [ 385.394341] Key type .llcrypt unregistered [ 385.395469] Key type ._llcrypt unregistered [ 387.258765] Key type ._llcrypt registered [ 387.260080] Key type .llcrypt registered [ 387.321924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 387.326858] alg: No test for adler32 (adler32-zlib) [ 388.125154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 388.134278] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 390.911522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 394.151028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.162229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 394.169209] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 394.574294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_interrupt -r 1 [ 394.587304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_interrupt -r 1 [ 394.600272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_interrupt -r 1 [ 394.613534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_interrupt -r 1 [ 394.625349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_interrupt -r 1 [ 394.638775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_interrupt -r 1 [ 394.652350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_interrupt -r 1 [ 394.663816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_interrupt -r 1 [ 394.675615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 394.682606] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 394.685116] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 394.753682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 394.763476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 394.772991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 394.784137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 394.794612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 394.805168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 394.815796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 394.826268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 394.833125] LNet: 57480:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 394.840093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 394.850486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 394.861206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 394.871707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 394.882765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 394.893794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 394.905272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 395.067750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_dropped -r 1 [ 395.081634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_dropped -r 1 [ 395.093560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_dropped -r 1 [ 395.105589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_dropped -r 1 [ 395.117376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_dropped -r 1 [ 395.129560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_dropped -r 1 [ 395.141584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_dropped -r 1 [ 395.153258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_dropped -r 1 [ 395.164229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 395.241732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 395.253377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 395.264718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 395.274926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 395.287177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 395.299082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 395.310286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 395.322466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 395.336218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 395.347096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 395.359060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 395.370449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 395.381524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 395.393575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 395.404952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 395.575975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_aborted -r 1 [ 395.588211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_aborted -r 1 [ 395.601716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_aborted -r 1 [ 395.615437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_aborted -r 1 [ 395.628393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_aborted -r 1 [ 395.641940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_aborted -r 1 [ 395.654357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_aborted -r 1 [ 395.667734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_aborted -r 1 [ 395.680907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 395.687580] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 395.689419] LNet: Skipped 6 previous similar messages [ 395.756862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 395.766596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 395.776066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 395.785423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 395.797388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 395.808042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 395.818466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 395.828453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 395.834545] LNet: 57722:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 395.836565] LNet: 57722:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 395.842439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 395.851753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 395.861791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 395.871082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 395.880784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 395.891861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 395.902259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 396.064959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_no_route -r 1 [ 396.076882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_no_route -r 1 [ 396.090606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_no_route -r 1 [ 396.104418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_no_route -r 1 [ 396.118363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_no_route -r 1 [ 396.132593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_no_route -r 1 [ 396.146719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_no_route -r 1 [ 396.160756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_no_route -r 1 [ 396.173721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 396.247468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 396.258416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 396.269957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 396.281115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 396.290514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 396.300191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 396.311959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 396.323191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 396.333466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 396.344609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 396.355474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 396.366927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 396.379418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 396.392377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 396.403504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 396.570935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_timeout -r 1 [ 396.584516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_timeout -r 1 [ 396.598618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_timeout -r 1 [ 396.611871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_timeout -r 1 [ 396.625777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_timeout -r 1 [ 396.638738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_timeout -r 1 [ 396.651616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_timeout -r 1 [ 396.664712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_timeout -r 1 [ 396.677746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 396.753752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 396.763367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 396.773125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 396.781832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 396.791307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 396.801185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 396.811426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 396.821252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 396.830721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 396.841529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 396.851268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 396.862706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 396.875930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 396.886233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 396.896954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 397.057205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_error -r 1 [ 397.071194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_error -r 1 [ 397.086136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_error -r 1 [ 397.098062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_error -r 1 [ 397.109529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_error -r 1 [ 397.120694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_error -r 1 [ 397.131967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_error -r 1 [ 397.142954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_error -r 1 [ 397.154399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 397.161110] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 397.163478] LNet: Skipped 8 previous similar messages [ 397.230538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 397.240929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 397.252221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 397.263640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 397.272994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 397.282077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 397.291552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 397.300678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 397.306234] LNet: 58085:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 397.308143] LNet: 58085:0:(api-ni.c:343:recovery_interval_set()) Skipped 2 previous similar messages [ 397.315187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 397.330219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 397.344338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 397.357176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 397.369925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 397.381057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 397.391985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 399.484687] LNet: 58160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 399.487091] LNetError: 58160:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 399.489691] LNet: Removed LNI 192.168.202.3@tcp [ 400.549323] LNet: Removed LNI 192.168.202.3@tcp1 [ 400.687508] Key type .llcrypt unregistered [ 400.688919] Key type ._llcrypt unregistered [ 405.812574] Key type ._llcrypt registered [ 405.813819] Key type .llcrypt registered [ 406.287396] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:06:27 (1768140387) [ 406.581664] Key type .llcrypt unregistered [ 406.583040] Key type ._llcrypt unregistered [ 408.733213] Key type ._llcrypt registered [ 408.734855] Key type .llcrypt registered [ 408.798405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 408.805067] alg: No test for adler32 (adler32-zlib) [ 409.604657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 409.614666] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 412.418414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 412.628766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e remote_dropped -r 1 [ 412.641317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e remote_dropped -r 1 [ 412.657700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 412.665470] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 412.742391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 412.753480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 412.766180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 412.777042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 412.788656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 412.799161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 412.810288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 412.820760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 412.827194] LNet: 59399:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 412.833406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 412.843575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 412.853640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 412.862653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 412.872542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 412.885761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 412.897703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 413.079231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e remote_error -r 1 [ 413.094066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e remote_error -r 1 [ 413.110761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 413.190578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 413.202395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 413.215987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 413.229057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 413.243970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 413.255675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 413.268884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 413.284195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 413.297786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 413.309056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 413.323083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 413.335520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 413.349452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 413.364572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 413.377562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 413.549497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e remote_timeout -r 1 [ 413.562953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e remote_timeout -r 1 [ 413.575407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 413.582267] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 413.584747] LNet: Skipped 1 previous similar message [ 413.665332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 413.678794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 413.692828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 413.707710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 413.723391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 413.738925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 413.754483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 413.765434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 413.772203] LNet: 59617:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 413.774683] LNet: 59617:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 413.780828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 413.791535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 413.803050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 413.817036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 413.829418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 413.842864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 413.854995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 413.950702] LNet: 59657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.952852] LNetError: 59657:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 413.961341] LNet: Removed LNI 192.168.202.3@tcp [ 414.096338] Key type .llcrypt unregistered [ 414.097376] Key type ._llcrypt unregistered [ 419.167188] Key type ._llcrypt registered [ 419.168109] Key type .llcrypt registered [ 419.584289] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:06:40 (1768140400) [ 419.828301] Key type .llcrypt unregistered [ 419.829222] Key type ._llcrypt unregistered [ 421.658922] Key type ._llcrypt registered [ 421.660083] Key type .llcrypt registered [ 421.727588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 421.733093] alg: No test for adler32 (adler32-zlib) [ 422.540395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 422.551284] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 425.309693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 427.932414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.944477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 427.952784] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 428.411926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e remote_dropped -r 1 [ 428.428364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e remote_dropped -r 1 [ 428.440459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e remote_dropped -r 1 [ 428.452412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e remote_dropped -r 1 [ 428.464395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e remote_dropped -r 1 [ 428.476435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e remote_dropped -r 1 [ 428.488517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e remote_dropped -r 1 [ 428.502142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e remote_dropped -r 1 [ 428.514291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 428.521600] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 428.523501] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.103@tcp [ 428.599524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 428.610826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 428.622658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 428.634260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 428.645129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 428.655682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 428.666288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 428.678267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 428.688331] LNet: 61114:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 428.696974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 428.707836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 428.718844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 428.728949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 428.739664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 428.752686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 428.764269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 428.773886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 428.947562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e remote_error -r 1 [ 428.961356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e remote_error -r 1 [ 428.973601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e remote_error -r 1 [ 428.986492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e remote_error -r 1 [ 429.001851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e remote_error -r 1 [ 429.014477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e remote_error -r 1 [ 429.028147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e remote_error -r 1 [ 429.041696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e remote_error -r 1 [ 429.055672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 429.062448] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 429.064374] LNet: Skipped 2 previous similar messages [ 429.139032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 429.152675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 429.165702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 429.177642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 429.189301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 429.200165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 429.215105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 429.232283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 429.242729] LNet: 61237:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 429.253050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 429.268313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 429.280813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 429.295593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 429.306698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 429.318045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 429.328846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 429.339331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 429.509044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e remote_timeout -r 1 [ 429.523519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e remote_timeout -r 1 [ 429.538059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e remote_timeout -r 1 [ 429.551939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e remote_timeout -r 1 [ 429.564401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e remote_timeout -r 1 [ 429.581954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e remote_timeout -r 1 [ 429.599590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e remote_timeout -r 1 [ 429.617886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e remote_timeout -r 1 [ 429.635584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 429.712586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 429.722957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 429.733521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 429.743546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 429.754017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 429.765042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 429.775931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 429.788202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 429.803289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 429.815522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 429.824922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 429.834679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 429.844377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 429.854953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 429.865669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 429.878092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 431.992686] LNet: 61437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 431.994789] LNetError: 61437:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 431.997494] LNet: Removed LNI 192.168.202.3@tcp [ 432.003279] LNet: Removed LNI 192.168.202.3@tcp1 [ 432.147354] Key type .llcrypt unregistered [ 432.148294] Key type ._llcrypt unregistered [ 437.540825] Key type ._llcrypt registered [ 437.542404] Key type .llcrypt registered [ 437.997935] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:06:59 (1768140419) [ 438.158529] Key type .llcrypt unregistered [ 438.160442] Key type ._llcrypt unregistered [ 440.051116] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 440.093935] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 440.096111] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 440.097926] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 442.315613] Key type ._llcrypt registered [ 442.316786] Key type .llcrypt registered [ 442.374159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 442.380564] alg: No test for adler32 (adler32-zlib) [ 443.185718] LNet: 62506:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 443.190067] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 443.201533] LNet: 62520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 443.203767] LNetError: 62520:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 443.215295] LNet: Removed LNI 192.168.202.3@tcp [ 443.444350] Key type .llcrypt unregistered [ 443.446220] Key type ._llcrypt unregistered [ 445.446944] Key type ._llcrypt registered [ 445.448549] Key type .llcrypt registered [ 445.503457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 445.509203] alg: No test for adler32 (adler32-zlib) [ 446.317055] LNet: 62774:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 446.322191] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 446.323382] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 446.334594] LNet: 62788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 446.336655] LNetError: 62788:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 446.338994] LNet: Removed LNI 192.168.202.3@tcp [ 446.347379] LNet: Removed LNI 10.1.2.3@tcp1 [ 446.582420] Key type .llcrypt unregistered [ 446.583864] Key type ._llcrypt unregistered [ 448.449674] Key type ._llcrypt registered [ 448.451017] Key type .llcrypt registered [ 448.518312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 448.524305] alg: No test for adler32 (adler32-zlib) [ 449.330420] LNet: 63043:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 449.334961] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 449.336466] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 449.348791] LNet: 63057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 449.351342] LNetError: 63057:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 449.353824] LNet: Removed LNI 192.168.202.3@tcp [ 449.359284] LNet: Removed LNI 10.1.2.3@tcp [ 449.610505] Key type .llcrypt unregistered [ 449.612446] Key type ._llcrypt unregistered [ 451.513095] Key type ._llcrypt registered [ 451.514045] Key type .llcrypt registered [ 451.562454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 451.567273] alg: No test for adler32 (adler32-zlib) [ 452.365712] LNet: 63313:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 452.369847] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 452.371296] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 452.382661] LNet: 63327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 452.384785] LNetError: 63327:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 452.386740] LNet: Removed LNI 192.168.202.3@tcp [ 452.398277] LNet: Removed LNI 10.1.2.3@tcp [ 452.613295] Key type .llcrypt unregistered [ 452.614290] Key type ._llcrypt unregistered [ 454.477551] Key type ._llcrypt registered [ 454.478542] Key type .llcrypt registered [ 454.535603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 454.540501] alg: No test for adler32 (adler32-zlib) [ 455.346053] LNet: 63586:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 455.352318] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 455.353737] LNetError: 63586:0:(config.c:1560:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 455.356869] LNetError: Error -100 starting up LNI tcp [ 455.365330] LNet: Removed LNI 192.168.202.3@tcp [ 455.618412] Key type .llcrypt unregistered [ 455.620389] Key type ._llcrypt unregistered [ 459.282798] Key type ._llcrypt registered [ 459.284317] Key type .llcrypt registered [ 459.775839] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:07:20 (1768140440) [ 460.050309] Key type .llcrypt unregistered [ 460.051475] Key type ._llcrypt unregistered [ 462.079314] Key type ._llcrypt registered [ 462.080242] Key type .llcrypt registered [ 462.143230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 462.150602] alg: No test for adler32 (adler32-zlib) [ 462.957338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 462.969162] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 465.798514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 466.002381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e network_timeout -r 1 [ 466.015672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e network_timeout -r 1 [ 466.028126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 466.036419] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 466.079100] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 466.103165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 466.115133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 466.125343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 466.135517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 466.145041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 466.155362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 466.164921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 466.174688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 466.180131] LNet: 64832:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 466.185948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 466.196242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 466.206307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 466.217792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 466.228492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 466.240803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 466.328821] LNet: 64870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 466.331300] LNetError: 64870:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 466.338322] LNet: Removed LNI 192.168.202.3@tcp [ 466.463393] Key type .llcrypt unregistered [ 466.464375] Key type ._llcrypt unregistered [ 472.018539] Key type ._llcrypt registered [ 472.020035] Key type .llcrypt registered [ 472.083321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 472.088145] alg: No test for adler32 (adler32-zlib) [ 472.890066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 472.901518] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 475.696052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 478.384161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.396743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 478.405614] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 478.869370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e network_timeout -r 1 [ 478.884292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e network_timeout -r 1 [ 478.898368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e network_timeout -r 1 [ 478.911063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e network_timeout -r 1 [ 478.924336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e network_timeout -r 1 [ 478.937235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e network_timeout -r 1 [ 478.951027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e network_timeout -r 1 [ 478.964119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e network_timeout -r 1 [ 478.977111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 478.984253] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 479.061756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 479.071402] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 479.073638] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.103@tcp [ 479.074388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 479.087899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 479.101750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 479.114703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 479.125879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 479.137992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 479.149415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 479.156415] LNet: 65740:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 479.163816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 479.177691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 479.189691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 479.200735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 479.213505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 479.228524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 481.186390] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 481.189081] LNet: Skipped 2 previous similar messages [ 481.288094] LNet: 65812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 481.291112] LNetError: 65812:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 481.293914] LNet: Removed LNI 192.168.202.3@tcp [ 481.300338] LNet: Removed LNI 192.168.202.3@tcp1 [ 481.445448] Key type .llcrypt unregistered [ 481.446849] Key type ._llcrypt unregistered [ 486.669259] Key type ._llcrypt registered [ 486.670231] Key type .llcrypt registered [ 487.081564] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:07:48 (1768140468) [ 487.193310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 487.197841] alg: No test for adler32 (adler32-zlib) [ 487.996576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.015879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 488.023934] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 488.246566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 488.254710] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 488.533639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 488.558486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 488.573125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 488.586325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 488.598508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 488.611872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 488.624583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.3@tcp [ 488.632275] LNetError: 66906:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.202.3@tcp to recovery [ 489.055263] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 489.055292] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 491.103343] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 491.785837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 495.199367] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 495.959940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 495.972919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 495.995787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.003128] LNet: 67023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 496.005922] LNetError: 67023:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 496.008799] LNet: Removed LNI 192.168.202.3@tcp [ 496.017312] LNet: Removed LNI 192.168.202.3@tcp1 [ 496.025570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.043659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 496.053094] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 496.297171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 496.304815] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 496.593756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 496.606121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 496.626726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 496.643098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 496.657143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 496.669231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 496.680169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 496.691498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.3@tcp [ 496.698786] LNetError: 67295:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.202.3@tcp to recovery [ 499.103412] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 499.105092] LNet: Skipped 1 previous similar message [ 499.847676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 503.199310] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 508.130796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 508.141613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 508.152611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 510.217552] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 09:08:11 (1768140491) [ 510.257466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.265319] LNet: 68019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 510.267952] LNetError: 68019:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 510.270231] LNetError: 68019:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 510.272464] LNet: Removed LNI 192.168.202.3@tcp [ 510.282339] LNet: Removed LNI 192.168.202.3@tcp1 [ 510.290709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.312241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 510.321200] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 510.548089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 510.556050] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 510.832737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 510.853783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 510.866035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 510.880670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 510.892977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 510.904364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 510.915325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.3@tcp --health 0 [ 510.947564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 511.327255] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.3@tcp [ 511.327289] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 511.330604] LNet: Skipped 1 previous similar message [ 526.330855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 526.384141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 526.397432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 526.408067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.3@tcp --health 500 [ 526.440615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 533.648090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 533.667982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.675066] LNet: 68529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 533.677117] LNetError: 68529:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 533.678983] LNetError: 68529:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 533.681048] LNet: Removed LNI 192.168.202.3@tcp [ 533.692285] LNet: Removed LNI 192.168.202.3@tcp1 [ 533.699397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.716571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 533.724143] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 533.954767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 533.963112] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 534.267985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 534.281729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 534.307557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 534.323318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 534.336981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 534.351710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 534.364690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 534.377214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.3@tcp --health 0 [ 535.775266] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 535.777997] LNet: Skipped 3 previous similar messages [ 536.527506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 546.910836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 546.923632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 546.935816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 549.159363] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 09:08:50 (1768140530) [ 549.297561] LNet: 69595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 549.300143] LNetError: 69595:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 549.303728] LNetError: 69595:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 549.305842] LNet: Removed LNI 192.168.202.3@tcp [ 549.310554] LNet: Removed LNI 192.168.202.3@tcp1 [ 549.496610] Key type .llcrypt unregistered [ 549.497520] Key type ._llcrypt unregistered [ 551.209237] Key type ._llcrypt registered [ 551.210370] Key type .llcrypt registered [ 551.270305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 551.275914] alg: No test for adler32 (adler32-zlib) [ 552.076740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 552.084744] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 555.250244] Lustre: DEBUG MARKER: Initial discovery [ 555.265420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.103@tcp [ 555.991614] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 556.022826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 556.038107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.103@tcp [ 556.046937] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 556.053708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 556.064905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.103@tcp -r 1 -m GET -l 3 [ 556.076622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.103@tcp -r 1 -m GET -e local_error [ 556.088703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.103@tcp -r 1 -m PUT -l 6 [ 556.191071] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 556.513139] Lustre: DEBUG MARKER: Force oleg203-client.virtnet to discover 192.168.202.103@tcp (in background) [ 556.542328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.103@tcp [ 556.956256] Lustre: DEBUG MARKER: Force oleg203-server to discover 192.168.202.3@tcp [ 557.665996] Lustre: DEBUG MARKER: Wait for 70346 [ 559.583174] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 560.000870] Lustre: DEBUG MARKER: Finished wait on 70346 [ 560.029750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 560.044318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 560.120363] LNet: 70799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 560.122262] LNetError: 70799:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 560.129283] LNet: Removed LNI 192.168.202.3@tcp [ 560.243289] Key type .llcrypt unregistered [ 560.244239] Key type ._llcrypt unregistered [ 564.664209] Key type ._llcrypt registered [ 564.665279] Key type .llcrypt registered [ 565.133443] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 09:09:06 (1768140546) [ 565.301250] Key type .llcrypt unregistered [ 565.303222] Key type ._llcrypt unregistered [ 567.183083] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 567.229799] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 567.233189] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 567.234839] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 569.441110] Key type ._llcrypt registered [ 569.442398] Key type .llcrypt registered [ 569.512583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 569.519219] alg: No test for adler32 (adler32-zlib) [ 570.322142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.342789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 570.351800] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 570.606669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 570.614367] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 573.297911] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 09:09:14 (1768140554) [ 573.338444] LNet: 72863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 573.341348] LNetError: 72863:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 573.343949] LNet: Removed LNI 192.168.202.3@tcp [ 573.349683] LNet: Removed LNI 10.1.2.3@tcp [ 573.559369] Key type .llcrypt unregistered [ 573.560584] Key type ._llcrypt unregistered [ 575.440956] Key type ._llcrypt registered [ 575.442248] Key type .llcrypt registered [ 575.501873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 575.506757] alg: No test for adler32 (adler32-zlib) [ 576.308208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.329640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 576.337938] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 576.589175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 576.600741] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 576.931451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.3@tcp [ 577.951087] LNetError: Unexpected error -22 connecting to 192.168.202.3@tcp at host 192.168.202.3:988 [ 578.953675] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 09:09:20 (1768140560) [ 579.001864] LNet: 73988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 579.006558] LNetError: 73988:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 579.010604] LNet: Removed LNI 192.168.202.3@tcp [ 579.018366] LNet: Removed LNI 10.1.2.3@tcp [ 579.237508] Key type .llcrypt unregistered [ 579.238932] Key type ._llcrypt unregistered [ 581.128981] Key type ._llcrypt registered [ 581.130477] Key type .llcrypt registered [ 581.210776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 581.218720] alg: No test for adler32 (adler32-zlib) [ 582.020819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.038988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 582.047368] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 582.302206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 582.309770] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 582.596761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.3@tcp1 --nid 192.168.202.3@tcp2 [ 583.984520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp1 192.168.202.3@tcp2 [ 583.992595] LNetError: 74823:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.3@tcp1 to 192.168.202.3@tcp2 - Route table may be misconfigured [ 584.000697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp2 192.168.202.3@tcp1 [ 586.099091] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 09:09:27 (1768140567) [ 586.133261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.141712] LNet: 75392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 586.143776] LNetError: 75392:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 586.146108] LNet: Removed LNI 192.168.202.3@tcp1 [ 587.173621] LNet: Removed LNI 192.168.202.3@tcp2 [ 587.182345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.203573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 587.213226] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 587.453831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 587.463568] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 587.765307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 587.782946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp 192.168.202.3@tcp [ 587.796637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp1 192.168.202.3@tcp1 [ 587.812162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.3@tcp -d 192.168.202.3@tcp -e network_timeout [ 587.827339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -e network_timeout [ 587.842315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -e network_timeout [ 587.859067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -e network_timeout [ 587.874109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 587.880948] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 587.888970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 587.925609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 587.960571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 588.255272] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 590.065137] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:09:31 (1768140571) [ 590.103887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.113916] LNet: 76252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.116465] LNetError: 76252:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 590.120038] LNetError: 76252:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 590.124653] LNet: Removed LNI 192.168.202.3@tcp [ 591.141576] LNet: Removed LNI 192.168.202.3@tcp1 [ 591.150152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.183786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 591.270740] LNet: 76293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 591.413667] Key type .llcrypt unregistered [ 591.414775] Key type ._llcrypt unregistered [ 594.957776] Key type ._llcrypt registered [ 594.958701] Key type .llcrypt registered [ 595.370078] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:09:36 (1768140576) [ 595.481525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 595.485982] alg: No test for adler32 (adler32-zlib) [ 596.289601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.755765] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 597.271073] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:09:38 (1768140578) [ 597.295461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.305534] LNet: 77381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.313744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.337123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 597.345380] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 597.561994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 597.569256] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 597.837658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 597.849216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp1 [ 597.860029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp1 [ 599.647702] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:09:40 (1768140580) [ 600.160724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg203-server [ 600.609697] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:09:41 (1768140581) [ 601.126679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg203-server [ 601.593842] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:09:42 (1768140582) [ 602.094711] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg203-server [ 602.568951] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:09:43 (1768140583) [ 603.090632] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg203-server [ 603.560444] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:09:44 (1768140584) [ 604.090925] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg203-server [ 604.589914] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:09:45 (1768140585) [ 605.124431] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg203-server [ 605.596468] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:09:46 (1768140586) [ 606.126130] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg203-server [ 606.619048] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:09:47 (1768140587) [ 607.151832] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg203-server [ 607.619612] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 09:09:48 (1768140588) [ 608.142028] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg203-server [ 608.610537] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:09:49 (1768140589) [ 608.666130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.672368] LNet: 81259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.674237] LNetError: 81259:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 608.675879] LNet: Removed LNI 192.168.202.3@tcp [ 609.701428] LNet: Removed LNI 192.168.202.3@tcp1 [ 609.707834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.724219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 609.733669] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 609.942534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 609.962650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 610.011644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.018646] LNet: 81406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.021425] LNetError: 81406:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 610.023572] LNetError: 81406:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 610.036282] LNet: Removed LNI 192.168.202.3@tcp [ 610.042951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.059564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 610.067388] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 610.290945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 610.339916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.365251] LNet: Removed LNI 192.168.202.3@tcp [ 611.371074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.385458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 611.391766] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 611.587448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 611.614760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 612.709078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.716134] LNet: 81707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.718747] LNet: 81707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 612.721690] LNetError: 81707:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 612.724093] LNetError: 81707:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 612.730279] LNet: Removed LNI 192.168.202.3@tcp [ 612.737416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.757637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 612.766546] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 613.005672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 613.064497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.117330] LNet: Removed LNI 192.168.202.3@tcp [ 614.125691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.145705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 614.155028] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 614.382961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 614.411340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 614.479043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.525343] LNet: Removed LNI 192.168.202.3@tcp [ 615.533848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.552574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 615.561930] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 615.819751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 615.891882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.900158] LNetError: 82145:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.902319] LNetError: 82145:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 615.910496] LNet: Removed LNI 192.168.202.3@tcp [ 615.917319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.938488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 615.948808] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 616.216944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 616.249761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 616.328324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.381315] LNet: Removed LNI 192.168.202.3@tcp [ 617.389690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.408451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 617.418223] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 617.699963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 617.771845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.780404] LNet: 82437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.783926] LNet: 82437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 618.857343] LNet: Removed LNI 192.168.202.3@tcp [ 618.864365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.882026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 618.890296] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 619.128168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 619.155581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 619.237354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.261333] LNet: Removed LNI 192.168.202.3@tcp [ 620.269837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.291438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 620.302126] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 620.556479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 621.677962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.685038] LNetError: 82739:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.687260] LNetError: 82739:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 621.696506] LNet: Removed LNI 192.168.202.3@tcp [ 621.704490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.726703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 621.736053] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 621.966980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 621.992730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 622.067798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.145333] LNet: Removed LNI 192.168.202.3@tcp [ 623.153496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.173404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 623.182527] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 623.453181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 623.534358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.613300] LNet: Removed LNI 192.168.202.3@tcp [ 624.621234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.641806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 624.650737] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 624.881930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 624.901051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.920305] LNet: Removed LNI 192.168.202.3@tcp [ 624.926834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.949099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 624.957339] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 625.206153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 627.319628] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:10:08 (1768140608) [ 627.361717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.370436] LNet: 83865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.372556] LNet: 83865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 627.386048] LNet: Removed LNI 192.168.202.3@tcp [ 627.394173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.414827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 627.423909] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/180] [ 627.809550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.825259] LNet: Removed LNI 192.168.202.3@tcp231 [ 627.833759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.857580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 627.868779] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/180] [ 627.870914] LNet: 84013:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 628.011889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 628.029675] LNet: Removed LNI 192.168.202.3@tcp231 [ 628.036215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 628.046090] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/180] [ 628.421707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 628.436309] LNet: Removed LNI 192.168.202.3@tcp231 [ 628.444262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.467147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 628.477337] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/0] [ 628.480075] LNet: 84165:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 628.616182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 628.633485] LNet: Removed LNI 192.168.202.3@tcp231 [ 628.639755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 628.650042] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/0] [ 629.049323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.063928] LNet: Removed LNI 192.168.202.3@tcp231 [ 629.071733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.098054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 629.107853] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/60] [ 629.251797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 629.270702] LNet: Removed LNI 192.168.202.3@tcp231 [ 629.278380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 629.288463] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/60] [ 629.699991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.707583] LNetError: 84459:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.709721] LNetError: 84459:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 629.718333] LNet: Removed LNI 192.168.202.3@tcp231 [ 629.726132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.750024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 629.758146] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/180] [ 629.760436] LNet: 84467:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 629.762832] LNet: 84467:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 629.896916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 629.917532] LNet: Removed LNI 192.168.202.3@tcp231 [ 629.923808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 629.932211] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/180] [ 632.328701] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:10:13 (1768140613) [ 632.363247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.379829] LNet: Removed LNI 192.168.202.3@tcp231 [ 632.390615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.418398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 632.430182] LNet: Added LNI 192.168.202.3@tcp232 [8/256/0/180] [ 632.699588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 632.718988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 632.738341] LNet: Removed LNI 192.168.202.3@tcp232 [ 632.746485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 632.757170] LNet: Added LNI 192.168.202.3@tcp232 [8/256/0/180] [ 633.017140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 633.033678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 633.049325] LNet: Removed LNI 192.168.202.3@tcp232 [ 633.056201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 633.065324] LNet: Added LNI 192.168.202.3@tcp232 [8/256/0/180] [ 633.325271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 633.344030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 633.361289] LNet: Removed LNI 192.168.202.3@tcp232 [ 633.369018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 633.378686] LNet: Added LNI 192.168.202.3@tcp232 [8/256/0/180] [ 633.656675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 633.674085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 633.693400] LNet: Removed LNI 192.168.202.3@tcp232 [ 633.701317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 635.701687] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 09:10:16 (1768140616) [ 636.065772] Key type .llcrypt unregistered [ 636.067177] Key type ._llcrypt unregistered [ 638.079295] Key type ._llcrypt registered [ 638.080794] Key type .llcrypt registered [ 638.145319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 638.151128] alg: No test for adler32 (adler32-zlib) [ 638.955180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 638.965022] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 641.850957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 644.430901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.442061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 644.448842] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 644.905637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_interrupt -r 2 [ 644.917642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_interrupt -r 2 [ 644.929243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_interrupt -r 2 [ 644.941201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_interrupt -r 2 [ 644.955227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_interrupt -r 2 [ 644.971671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_interrupt -r 2 [ 644.984877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_interrupt -r 2 [ 645.000314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_interrupt -r 2 [ 645.015109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp 192.168.202.103@tcp [ 645.023264] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 645.026494] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 645.100624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 645.112849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 645.125151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 645.135976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 645.147900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 645.160798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 645.171092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 645.183475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 645.190966] LNet: 86959:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 645.198194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 645.208627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 645.219764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 645.231403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 645.242994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 645.257323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 645.430910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_dropped -r 2 [ 645.444154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_dropped -r 2 [ 645.461320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_dropped -r 2 [ 645.475842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_dropped -r 2 [ 645.488900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_dropped -r 2 [ 645.501507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_dropped -r 2 [ 645.513937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_dropped -r 2 [ 645.526675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_dropped -r 2 [ 645.541549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp 192.168.202.103@tcp [ 645.552849] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 645.637988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 645.649815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 645.660764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 645.671495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 645.681868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 645.692625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 645.704593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 645.716451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 645.723403] LNet: 87079:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 645.730456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 645.742447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 645.754312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 645.765587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 645.777272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 645.789323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 645.963631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_aborted -r 2 [ 645.978745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_aborted -r 2 [ 645.993589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_aborted -r 2 [ 646.007829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_aborted -r 2 [ 646.020873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_aborted -r 2 [ 646.036053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_aborted -r 2 [ 646.051704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_aborted -r 2 [ 646.068864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_aborted -r 2 [ 646.085395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp 192.168.202.103@tcp [ 646.169636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 646.182221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 646.192611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 646.202116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 646.212462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 646.221823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 646.230996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 646.241422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 646.252155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 646.262093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 646.273286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 646.286879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 646.300398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 646.313563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 646.490902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_no_route -r 2 [ 646.504048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_no_route -r 2 [ 646.518409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_no_route -r 2 [ 646.532867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_no_route -r 2 [ 646.545661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_no_route -r 2 [ 646.562995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_no_route -r 2 [ 646.576262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_no_route -r 2 [ 646.591444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_no_route -r 2 [ 646.604935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp 192.168.202.103@tcp [ 646.612838] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 646.615070] LNet: Skipped 2 previous similar messages [ 646.696483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 646.709520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 646.726397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 646.739281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 646.749431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 646.759472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 646.770337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 646.784563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 646.791532] LNet: 87319:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 646.794092] LNet: 87319:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 646.803902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 646.816281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 646.827537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 646.840715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 646.853119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 646.864554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 647.062557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_timeout -r 2 [ 647.079872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_timeout -r 2 [ 647.095984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_timeout -r 2 [ 647.111229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_timeout -r 2 [ 647.125553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_timeout -r 2 [ 647.144618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_timeout -r 2 [ 647.157873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_timeout -r 2 [ 647.171131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_timeout -r 2 [ 647.184066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp 192.168.202.103@tcp [ 647.289550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 647.299959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 647.310347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 647.321440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 647.332525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 647.344354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 647.356306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 647.367640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 647.379675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 647.391430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 647.402903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 647.414090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 647.424752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 647.435577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 647.607747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e remote_dropped -r 2 [ 647.623705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e remote_dropped -r 2 [ 647.638592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e remote_dropped -r 2 [ 647.652638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e remote_dropped -r 2 [ 647.667312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e remote_dropped -r 2 [ 647.681318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e remote_dropped -r 2 [ 647.696234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e remote_dropped -r 2 [ 647.712495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e remote_dropped -r 2 [ 647.727300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp 192.168.202.103@tcp [ 647.736775] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.103@tcp [ 647.811843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 647.823231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 647.835062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 647.846216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 647.856869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 647.870853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 647.886529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 647.899533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 647.911595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 647.924972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 647.935828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 647.946610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 647.959365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 647.970396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 649.976889] LNet: 87632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.979243] LNetError: 87632:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 649.981488] LNet: Removed LNI 192.168.202.3@tcp [ 649.988325] LNet: Removed LNI 192.168.202.3@tcp1 [ 650.119474] Key type .llcrypt unregistered [ 650.120579] Key type ._llcrypt unregistered [ 655.448899] Key type ._llcrypt registered [ 655.450095] Key type .llcrypt registered [ 655.894117] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 09:10:36 (1768140636) [ 656.020256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 656.026878] alg: No test for adler32 (adler32-zlib) [ 656.828366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.848559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 656.858535] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 657.133782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 657.147361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.3@tcp [ 657.154571] LNetError: 88605:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.202.3@tcp to recovery [ 657.182126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.3@tcp [ 657.213112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 659.216097] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 09:10:40 (1768140640) [ 659.251647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.261773] LNet: 89190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.265365] LNetError: 89190:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 659.275372] LNet: Removed LNI 192.168.202.3@tcp [ 659.283629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.306379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 659.314954] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 659.558360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 659.567177] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 659.916673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.3@tcp [ 659.923617] LNetError: 89460:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.202.3@tcp to recovery [ 660.319337] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 660.319942] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 661.000444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.3@tcp [ 665.252766] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 09:10:46 (1768140646) [ 665.876279] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg203-server [ 666.433230] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 09:10:47 (1768140647) [ 666.469177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.481373] LNet: 90412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.485373] LNetError: 90412:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 666.489050] LNet: Removed LNI 192.168.202.3@tcp [ 666.496840] LNet: Removed LNI 192.168.202.3@tcp1 [ 666.506045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.525646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 666.538228] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 666.800584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 666.828168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 666.848348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 666.873960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 666.892166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 666.908877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 668.923620] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 09:10:49 (1768140649) [ 668.967363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.976689] LNet: 91142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.978785] LNetError: 91142:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.981333] LNetError: 91142:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 668.988923] LNet: Removed LNI 192.168.202.3@tcp [ 668.998317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.020225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 669.029940] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 671.153326] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 09:10:52 (1768140652) [ 671.194023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.204194] LNet: 91844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.206217] LNetError: 91844:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.214637] LNet: Removed LNI 192.168.202.3@tcp [ 671.222858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.241536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 671.249764] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 673.470609] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 09:10:54 (1768140654) [ 673.950629] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 674.512604] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 09:10:55 (1768140655) [ 674.656603] LNet: Removed LNI 192.168.202.3@tcp [ 674.864552] Key type .llcrypt unregistered [ 674.865923] Key type ._llcrypt unregistered [ 676.874335] Key type ._llcrypt registered [ 676.875484] Key type .llcrypt registered [ 676.942726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 676.949376] alg: No test for adler32 (adler32-zlib) [ 677.752426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 677.764505] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 680.603738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 682.132286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.103@tcp [ 682.253288] LNet: 93289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.257730] LNetError: 93289:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 682.270431] LNet: Removed LNI 192.168.202.3@tcp [ 682.420479] Key type .llcrypt unregistered [ 682.422059] Key type ._llcrypt unregistered [ 686.189996] Key type ._llcrypt registered [ 686.191096] Key type .llcrypt registered [ 686.640074] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 687.107592] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 687.631459] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 09:11:08 (1768140668) [ 687.747859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 687.754292] alg: No test for adler32 (adler32-zlib) [ 688.557158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.585373] LNet: 94372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.784306] Key type .llcrypt unregistered [ 688.785587] Key type ._llcrypt unregistered [ 690.682864] Key type ._llcrypt registered [ 690.684036] Key type .llcrypt registered [ 690.744058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 690.751680] alg: No test for adler32 (adler32-zlib) [ 691.568262] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 691.570319] LNet: 94623:0:(config.c:1226:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 691.574238] LNet: 94623:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 691.783144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.951344] LNetError: 94635:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to o2ib through 192.168.202.7@tcp has gone from up to down [ 697.956259] LNet: 94681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.959427] LNetError: 94681:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 697.975651] LNet: Removed LNI 192.168.202.3@tcp [ 697.988557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.017345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.026428] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 698.029640] LNet: 94689:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 698.032360] LNet: 94689:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 698.035262] LNetError: 94689:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.202.4@tcp is a gateway. Can not delete it [ 698.038576] LNetError: 94689:0:(peer.c:2159:lnet_del_peer_ni()) 192.168.202.4@tcp is a router. Can not be deleted [ 698.043221] LNet: 94689:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 700.281455] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 09:11:21 (1768140681) [ 700.831337] LNetError: 94691:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to o2ib through 192.168.202.8@tcp has gone from up to down [ 700.834655] LNetError: 94691:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 700.898231] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg203-server [ 701.485595] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 09:11:22 (1768140682) [ 702.104616] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg203-server [ 702.702664] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 09:11:23 (1768140683) [ 703.441798] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 703.903431] LNetError: 94691:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to o2ib through 192.168.202.5@tcp has gone from up to down [ 703.907850] LNetError: 94691:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 705.119127] Key type lgssc registered [ 705.514469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 713.334948] Key type lgssc unregistered [ 713.491667] LNet: 97104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.493817] LNetError: 97104:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 713.503829] LNet: Removed LNI 192.168.202.3@tcp [ 713.738569] Key type .llcrypt unregistered [ 713.740451] Key type ._llcrypt unregistered [ 718.616637] Key type ._llcrypt registered [ 718.617535] Key type .llcrypt registered [ 719.081258] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 09:11:40 (1768140700) [ 719.214573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 719.221254] alg: No test for adler32 (adler32-zlib) [ 720.021751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.048966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 720.065895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 720.083828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 720.103753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 722.200921] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 09:11:43 (1768140703) [ 722.249307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.258362] LNet: 98503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.269921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.294391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 722.304380] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 722.611961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.620363] LNetError: 98648:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.630268] LNet: Removed LNI 192.168.202.3@tcp [ 722.639306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.659389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 722.668932] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 722.829601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.837673] LNet: 98676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.839602] LNet: 98676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 722.848336] LNet: Removed LNI 192.168.202.3@tcp [ 722.856200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.879159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 722.890056] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 725.162385] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 09:11:46 (1768140706) [ 725.194527] LNet: 99259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.197335] LNetError: 99259:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.199586] LNetError: 99259:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 725.206577] LNet: Removed LNI 192.168.202.3@tcp [ 725.450478] Key type .llcrypt unregistered [ 725.451665] Key type ._llcrypt unregistered [ 727.634097] Key type ._llcrypt registered [ 727.634962] Key type .llcrypt registered [ 727.702864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 727.708210] alg: No test for adler32 (adler32-zlib) [ 728.521829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.549312] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 728.550550] LNet: *** cfs_fail_loc=e003, val=100*** [ 728.848699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 728.856654] Ping reply received with mapping: [ 728.857859] found metadata reply NI[192.168.202.3@tcp] [ 728.859380] 03 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 728.860715] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 728.862183] 5a 5a 5a 5a 5a 5a 5a 5a [ 728.914413] LNet: 99659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.916824] LNetError: 99659:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.930320] LNet: Removed LNI 192.168.202.3@tcp [ 729.182490] Key type .llcrypt unregistered [ 729.183678] Key type ._llcrypt unregistered [ 732.885456] Key type ._llcrypt registered [ 732.887175] Key type .llcrypt registered [ 733.381898] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 09:11:54 (1768140714) [ 733.559370] Key type .llcrypt unregistered [ 733.560466] Key type ._llcrypt unregistered [ 735.634228] Key type ._llcrypt registered [ 735.635180] Key type .llcrypt registered [ 735.699181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 735.706233] alg: No test for adler32 (adler32-zlib) [ 736.571360] LNetError: 100693:0:(api-ni.c:2732:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 736.720297] Key type .llcrypt unregistered [ 736.721809] Key type ._llcrypt unregistered [ 738.516211] Key type ._llcrypt registered [ 738.517135] Key type .llcrypt registered [ 738.965336] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 09:11:59 (1768140719) [ 739.154584] Key type .llcrypt unregistered [ 739.156519] Key type ._llcrypt unregistered [ 741.185480] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 741.188065] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 743.347636] Key type ._llcrypt registered [ 743.348656] Key type .llcrypt registered [ 743.409615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 743.414662] alg: No test for adler32 (adler32-zlib) [ 744.213212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.234930] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 747.946686] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 09:12:09 (1768140729) [ 747.980237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.990693] LNet: 102460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 747.993278] LNetError: 102460:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 748.001955] LNet: Removed LNI 10.1.2.3@tcp [ 748.014109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.276367] Key type .llcrypt unregistered [ 748.277280] Key type ._llcrypt unregistered [ 750.134059] Key type ._llcrypt registered [ 750.135049] Key type .llcrypt registered [ 750.194204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 750.200142] alg: No test for adler32 (adler32-zlib) [ 751.009574] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 751.030459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 751.081206] LNet: 102747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.084381] LNetError: 102747:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 751.092367] LNet: Removed LNI 192.168.202.3@tcp [ 751.309314] Key type .llcrypt unregistered [ 751.310244] Key type ._llcrypt unregistered [ 754.967343] Key type ._llcrypt registered [ 754.969070] Key type .llcrypt registered [ 755.444637] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 09:12:16 (1768140736) [ 755.655469] Key type .llcrypt unregistered [ 755.656839] Key type ._llcrypt unregistered [ 757.796270] Key type ._llcrypt registered [ 757.797284] Key type .llcrypt registered [ 757.861522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 757.868083] alg: No test for adler32 (adler32-zlib) [ 760.870913] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 09:12:21 (1768140741) [ 760.907342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.922206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.954724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 760.963553] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 761.219746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 761.233050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 761.250563] LNet: Removed LNI 192.168.202.3@tcp [ 761.325383] LNet: 104480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.471303] Key type .llcrypt unregistered [ 761.472478] Key type ._llcrypt unregistered [ 765.173616] Key type ._llcrypt registered [ 765.174476] Key type .llcrypt registered [ 765.637143] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 09:12:26 (1768140746) [ 765.760126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 765.767877] alg: No test for adler32 (adler32-zlib) [ 766.569580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.591321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 766.600109] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 766.835854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 766.848434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.3@tcp [ 766.854857] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 766.856257] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 766.857374] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 766, 0, 0 [ 766.859056] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 766.859988] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 766.861181] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 766.862282] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 766, 0, 0 [ 766.864112] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 766.865131] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 766.866446] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 766.867604] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 766, 0, 816 [ 766.869248] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 766.870188] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 766.871460] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 766.872695] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 766, 0, 816 [ 766.874197] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 768.867640] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 09:12:29 (1768140749) [ 769.002616] LNet: 106015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.004856] LNetError: 106015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.012534] LNet: Removed LNI 192.168.202.3@tcp [ 769.226599] Key type .llcrypt unregistered [ 769.227815] Key type ._llcrypt unregistered [ 771.238617] Key type ._llcrypt registered [ 771.239936] Key type .llcrypt registered [ 771.304716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 771.310636] alg: No test for adler32 (adler32-zlib) [ 772.118981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 772.131734] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 774.938816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 777.524849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.536972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 777.544661] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 777.867466] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 777.915684] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 777.917222] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 777.919020] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 780.058506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 780.067848] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 783.388236] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:12:44 (1768140764) [ 783.446581] LNet: 107512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.449601] LNetError: 107512:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.452377] LNet: Removed LNI 192.168.202.3@tcp [ 784.479127] LNet: Removed LNI 192.168.202.3@tcp1 [ 784.480287] LNetError: 107512:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 784.483853] LNetError: 107512:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 784.492492] LNet: Removed LNI 10.1.2.3@tcp99 [ 784.704402] Key type .llcrypt unregistered [ 784.705475] Key type ._llcrypt unregistered [ 786.620992] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 786.622690] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 788.809294] Key type ._llcrypt registered [ 788.810686] Key type .llcrypt registered [ 788.884929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 788.891101] alg: No test for adler32 (adler32-zlib) [ 789.696813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.721381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 789.732564] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 789.982687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 789.990669] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 790.746672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 790.766349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 790.785483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 790.793241] LNetError: 108253:0:(peer.c:2167:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 790.805736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 790.821743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 790.982013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 791.019491] LNet: 108273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.022847] LNetError: 108273:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.025794] LNet: Removed LNI 192.168.202.3@tcp [ 791.033546] LNet: Removed LNI 10.1.2.3@tcp [ 791.264448] Key type .llcrypt unregistered [ 791.265600] Key type ._llcrypt unregistered [ 794.958214] Key type ._llcrypt registered [ 794.959492] Key type .llcrypt registered [ 795.493867] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:12:56 (1768140776) [ 795.624247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 795.629503] alg: No test for adler32 (adler32-zlib) [ 796.435884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.457045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 796.467896] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 796.725522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 796.754642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg203-client@tcp [ 798.840195] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:12:59 (1768140779) [ 798.891736] LNet: 109799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.894554] LNetError: 109799:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.904629] LNet: Removed LNI 192.168.202.3@tcp [ 799.110386] Key type .llcrypt unregistered [ 799.111373] Key type ._llcrypt unregistered [ 800.929692] Key type ._llcrypt registered [ 800.930726] Key type .llcrypt registered [ 800.986524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 800.991300] alg: No test for adler32 (adler32-zlib) [ 801.788167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 801.797791] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 801.809530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 803.793569] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:13:04 (1768140784) [ 804.268819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 804.785910] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:13:05 (1768140785) [ 805.257298] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 805.771957] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 09:13:06 (1768140786) [ 805.818181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.831531] LNet: 111212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.833809] LNetError: 111212:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.841579] LNet: Removed LNI 192.168.202.3@tcp [ 805.848843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.867948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 805.876892] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 806.123573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 806.144169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 808.281243] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 09:13:09 (1768140789) [ 808.336709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.345920] LNet: 111921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.347852] LNetError: 111921:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.356514] LNet: Removed LNI 192.168.202.3@tcp [ 808.362467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.378399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 808.388504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 808.602314] Key type .llcrypt unregistered [ 808.603257] Key type ._llcrypt unregistered [ 812.048578] Key type ._llcrypt registered [ 812.049625] Key type .llcrypt registered [ 812.484846] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 09:13:13 (1768140793) [ 812.593359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 812.598170] alg: No test for adler32 (adler32-zlib) [ 813.397134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.418637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 813.426700] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 813.647601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 813.669902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 815.677467] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 09:13:16 (1768140796) [ 815.716850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.725666] LNet: 113468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.729781] LNetError: 113468:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.742693] LNet: Removed LNI 192.168.202.3@tcp [ 815.750648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.770630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 815.782437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 817.785978] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 09:13:18 (1768140798) [ 817.824379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.835714] LNet: 114047:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.850596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.874345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 817.994981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 818.005405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 818.026515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 818.167603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 818.178878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 818.203404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 818.215851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.239462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 818.365511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 818.378380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 818.398466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 818.505867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 818.516219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 820.444902] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 09:13:21 (1768140801) [ 820.488707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.496230] LNet: 114682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.499518] LNet: 114682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 820.510596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.528419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 820.537108] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 820.765848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 820.772640] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 821.026338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 821.043773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 821.112395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.119622] LNetError: 114962:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.122748] LNet: Removed LNI 192.168.202.3@tcp [ 821.132282] LNet: Removed LNI 192.168.202.3@tcp2 [ 821.140174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.160891] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 821.164140] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 821.165760] LNet: 114968:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 821.300770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 821.365865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 822.431134] LNet: Removed LNI 192.168.202.3@tcp [ 822.433104] LNetError: 115005:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 822.436458] LNetError: 115005:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 822.450314] LNet: Removed LNI 192.168.202.3@tcp2 [ 822.458425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.476301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 822.484897] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 822.726152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 822.735585] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 823.083706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 823.099823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 823.124725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 823.207120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 823.216095] LNet: 115289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.218262] LNet: 115289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 824.223139] LNet: Removed LNI 192.168.202.3@tcp [ 825.253326] LNet: Removed LNI 192.168.202.3@tcp2 [ 825.260891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.286906] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 825.288979] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 825.290892] LNet: 115295:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 825.437348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 825.520881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 825.529447] LNetError: 115332:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.531489] LNetError: 115332:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 825.533387] LNet: Removed LNI 192.168.202.3@tcp [ 826.597435] LNet: Removed LNI 192.168.202.3@tcp2 [ 826.605966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.628452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 826.638859] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 826.943249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 826.950132] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 827.262160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 827.286874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 827.371875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 827.380364] LNet: 115612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.382561] LNet: 115612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 828.447179] LNet: Removed LNI 192.168.202.3@tcp [ 828.454332] LNet: Removed LNI 192.168.202.3@tcp2 [ 828.462733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.492267] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 828.495293] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 828.497617] LNet: 115618:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 828.670976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 828.750805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.791249] LNet: Removed LNI 192.168.202.3@tcp [ 829.792641] LNetError: 115655:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.795374] LNetError: 115655:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 829.804513] LNet: Removed LNI 192.168.202.3@tcp2 [ 829.814528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.835287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 829.844237] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 830.135288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 830.145339] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 830.461418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 830.476954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 830.498371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 830.579226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 831.647183] LNet: Removed LNI 192.168.202.3@tcp [ 832.677412] LNet: Removed LNI 192.168.202.3@tcp2 [ 832.685058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.708672] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 832.710684] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 832.712497] LNet: 115945:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 832.871733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 832.944561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.954481] LNet: Removed LNI 192.168.202.3@tcp [ 834.021337] LNet: Removed LNI 192.168.202.3@tcp2 [ 834.030461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.054821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 834.065784] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 834.343615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 834.351961] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 834.686019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 834.699771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 834.720740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 834.874162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.935104] LNet: Removed LNI 192.168.202.3@tcp [ 835.942287] LNet: Removed LNI 192.168.202.3@tcp2 [ 835.949467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.972477] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 835.975026] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 836.139712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 836.253786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 836.260567] LNet: 116337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.262380] LNet: 116337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 837.279101] LNet: Removed LNI 192.168.202.3@tcp [ 837.290265] LNet: Removed LNI 192.168.202.3@tcp2 [ 837.297075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.315287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 837.323552] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 837.568455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 837.577230] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 837.872535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 837.886733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 837.897628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 837.917609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 838.035853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 838.042881] LNetError: 116638:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.045033] LNetError: 116638:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 838.046980] LNet: Removed LNI 192.168.202.3@tcp [ 839.077333] LNet: Removed LNI 192.168.202.3@tcp2 [ 839.085651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.111083] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 839.113888] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 839.116308] LNet: 116644:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 839.119286] LNet: 116644:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 839.291962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 841.558628] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 842.135720] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 09:13:43 (1768140823) [ 842.168723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.180554] LNet: Removed LNI 192.168.202.3@tcp [ 843.237869] LNet: Removed LNI 192.168.202.3@tcp2 [ 843.247101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.353406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 843.361894] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 843.628682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 843.636959] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 843.953949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.962606] LNet: Removed LNI 192.168.202.3@tcp [ 843.971374] LNet: Removed LNI 10.1.2.3@tcp [ 843.978719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.000736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 844.010612] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 844.012254] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 844.164265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 844.172909] LNet: Removed LNI 192.168.202.3@tcp [ 844.179635] LNet: Removed LNI 10.1.2.3@tcp [ 846.412428] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 09:13:47 (1768140827) [ 846.442203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.457205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.491751] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 846.531860] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 846.533672] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 846.535470] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 848.670391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 848.679160] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 848.928291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 848.939122] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 849.603665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 849.615849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 849.638845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.3@tcp -m PUT -e local_timeout -r 1 [ 849.663863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 849.683041] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 849.684483] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 849.765894] LNetError: 118339:0:(lib-move.c:4947:lnet_parse()) 192.168.202.3@tcp, src 192.168.202.3@tcp: Dropping PUT (error -108 looking up sender) [ 851.807165] LNet: Removed LNI 192.168.202.3@tcp [ 851.814284] LNet: Removed LNI 10.1.2.3@tcp [ 851.945630] Key type .llcrypt unregistered [ 851.946649] Key type ._llcrypt unregistered [ 855.643036] Key type ._llcrypt registered [ 855.643942] Key type .llcrypt registered [ 856.070482] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 09:13:57 (1768140837) [ 856.187714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 856.192803] alg: No test for adler32 (adler32-zlib) [ 856.994220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.138250] LNet: 119645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.382333] Key type .llcrypt unregistered [ 857.384125] Key type ._llcrypt unregistered [ 859.604107] Key type ._llcrypt registered [ 859.605266] Key type .llcrypt registered [ 859.672269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 859.678404] alg: No test for adler32 (adler32-zlib) [ 860.484241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 860.496704] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 863.299306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 863.347472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 863.530856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.103@tcp [ 863.584080] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.103@tcp [ 863.608661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 863.621509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 863.633075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 863.644523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 863.654422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 863.664866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 863.674749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 863.685298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 863.692596] LNet: 120089:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 863.701468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 863.714199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 863.727207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 863.738618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 863.749114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 863.759951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 863.964141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 864.045255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 864.055739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 864.065988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 864.076783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 864.087465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 864.098088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 864.108959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 864.120782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 864.133612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 864.143633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 864.154191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 864.165241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 864.177606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 864.190360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 864.400578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.103@tcp [ 864.473598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 864.483210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 864.493369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 864.506310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 864.517659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 864.527868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 864.538746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 864.548474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 864.555228] LNet: 120323:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 864.557186] LNet: 120323:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 864.562675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 864.572713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 864.582300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 864.594359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 864.608500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 864.624434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 864.818715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 864.889426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 864.899278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 864.909349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 864.919837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 864.930652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 864.941459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 864.952773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 864.963885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 864.974456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 864.985036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 864.996020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 865.006262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 865.016684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 865.028333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 865.119066] LNet: 120485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.121127] LNetError: 120485:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.134304] LNet: Removed LNI 192.168.202.3@tcp [ 865.277706] Key type .llcrypt unregistered [ 865.279101] Key type ._llcrypt unregistered [ 870.617988] Key type ._llcrypt registered [ 870.619616] Key type .llcrypt registered [ 871.055267] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 09:14:12 (1768140852) [ 871.160963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 871.166017] alg: No test for adler32 (adler32-zlib) [ 871.962907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.080764] LNet: 121367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.306257] Key type .llcrypt unregistered [ 872.307306] Key type ._llcrypt unregistered [ 874.217559] Key type ._llcrypt registered [ 874.218455] Key type .llcrypt registered [ 874.281349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 874.286750] alg: No test for adler32 (adler32-zlib) [ 875.090764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 875.099688] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 877.952111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 880.620549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.630346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 880.637745] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 880.955782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 881.135169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.103@tcp [ 881.216549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 881.225836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 881.234915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 881.244133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 881.247066] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.103@tcp [ 881.256544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 881.271526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 881.285393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 881.297072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 881.303675] LNet: 122012:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 881.309988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 881.320947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 881.330686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 881.343428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 881.356102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 881.369674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 881.585983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 881.669520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 881.680325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 881.693163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 881.708809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 881.724428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 881.738095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 881.752495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 881.764768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 881.776718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 881.788107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 881.800969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 881.814987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 881.830195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 881.845069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 882.067764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.103@tcp [ 882.146653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 882.157280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 882.168402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 882.177949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 882.189705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 882.200224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 882.211627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 882.222812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 882.229984] LNet: 122247:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 882.233133] LNet: 122247:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 882.240401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 882.252616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 882.265721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 882.275271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 882.286828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 882.298468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 882.502634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 882.585340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 882.595901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 882.605744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 882.616999] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 882.629514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 882.641371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 882.654896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 882.666664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 882.678825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 882.690457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 882.703165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 882.715083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 882.727583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 882.742521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 884.792684] LNet: 122443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.796428] LNetError: 122443:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.798806] LNet: Removed LNI 192.168.202.3@tcp [ 884.807334] LNet: Removed LNI 192.168.202.3@tcp1 [ 884.954557] Key type .llcrypt unregistered [ 884.956416] Key type ._llcrypt unregistered [ 890.300827] Key type ._llcrypt registered [ 890.301772] Key type .llcrypt registered [ 890.733538] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 09:14:31 (1768140871) [ 893.007264] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 09:14:34 (1768140874) [ 895.337938] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 09:14:36 (1768140876) [ 897.757294] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 09:14:38 (1768140878) [ 899.969935] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 09:14:41 (1768140881) [ 902.249821] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 09:14:43 (1768140883) [ 904.519751] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 09:14:45 (1768140885) [ 906.748366] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 09:14:47 (1768140887) [ 908.937239] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 09:14:50 (1768140890) [ 911.063818] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 09:14:52 (1768140892) [ 913.159233] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 09:14:54 (1768140894) [ 915.388804] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 09:14:56 (1768140896) [ 917.478685] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 09:14:58 (1768140898) [ 919.663030] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 09:15:00 (1768140900) [ 922.173352] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 09:15:03 (1768140903) [ 924.378616] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 09:15:05 (1768140905) [ 926.595838] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 09:15:07 (1768140907) [ 928.751230] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 09:15:09 (1768140909) [ 928.869719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 928.875846] alg: No test for adler32 (adler32-zlib) [ 929.675780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.716231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 929.723909] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 930.073757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.080550] LNet: 133134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.083097] LNetError: 133134:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.091327] LNet: Removed LNI 192.168.202.3@tcp [ 930.099989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.125592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 930.134601] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 932.178968] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 09:15:13 (1768140913) [ 934.405387] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 09:15:15 (1768140915) [ 936.570845] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 09:15:17 (1768140917) [ 938.712628] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 09:15:19 (1768140919) [ 940.804811] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 09:15:21 (1768140921) [ 943.015437] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 09:15:24 (1768140924) [ 945.366693] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 09:15:26 (1768140926) [ 947.660760] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 09:15:28 (1768140928) [ 949.917176] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 09:15:30 (1768140930) [ 952.289824] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 09:15:33 (1768140933) [ 954.502592] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 09:15:35 (1768140935) [ 956.824419] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 09:15:37 (1768140937) [ 958.914387] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 853 sec =========== 09:15:39 (1768140939) [ 959.023703] LNet: 140589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.026622] LNetError: 140589:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.037500] LNet: Removed LNI 192.168.202.3@tcp [ 959.243425] Key type .llcrypt unregistered [ 959.244424] Key type ._llcrypt unregistered [ 963.254250] Key type ._llcrypt registered [ 963.255373] Key type .llcrypt registered [ 963.419420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 963.430233] alg: No test for adler32 (adler32-zlib) [ 964.283687] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 964.364479] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 965.943204] Key type lgssc registered [ 966.313844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 987.749638] Lustre: Mounted lustre-client [ 990.095855] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 997.254845] LustreError: 143029:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffffa01d105c0000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 997.259638] LustreError: 143029:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 997.270400] Lustre: Unmounted lustre-client [ 1008.999664] Key type lgssc unregistered [ 1009.119519] LNet: 143510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.123315] LNetError: 143510:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.133328] LNet: Removed LNI 192.168.202.3@tcp [ 1009.398123] Key type .llcrypt unregistered [ 1009.399138] Key type ._llcrypt unregistered