[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 422750043 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002300] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006723] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007000] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007009] pid_max: default: 32768 minimum: 301 [ 0.008130] LSM: Security Framework initializing [ 0.009035] Yama: becoming mindful. [ 0.010024] SELinux: Initializing. [ 0.011047] *** VALIDATE selinux *** [ 0.024132] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029097] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031157] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033081] *** VALIDATE tmpfs *** [ 0.034000] *** VALIDATE proc *** [ 0.034229] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036282] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039264] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.043253] debug: unmapping init [mem 0xffffffff8ce59000-0xffffffff8ce60fff] [ 0.046336] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047655] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048018] ... version: 2 [ 0.049009] ... bit width: 48 [ 0.050009] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052051] ... max period: 00007fffffffffff [ 0.053019] ... fixed-purpose events: 3 [ 0.054046] ... event mask: 000000070000000f [ 0.055861] rcu: Hierarchical SRCU implementation. [ 0.058290] smp: Bringing up secondary CPUs ... [ 0.060787] x86: Booting SMP configuration: [ 0.061033] .... node #0, CPUs: #1 #2 #3 [ 0.065358] smp: Brought up 1 node, 4 CPUs [ 0.067020] smpboot: Max logical packages: 1 [ 0.068014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.191629] node 0 deferred pages initialised in 121ms [ 0.195207] devtmpfs: initialized [ 0.197363] x86/mm: Memory block size: 128MB [ 0.201049] gcov: version magic: 0x41383552 [ 0.203345] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.208166] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.210331] pinctrl core: initialized pinctrl subsystem [ 0.212226] [ 0.212793] ************************************************************* [ 0.214016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.217012] ** ** [ 0.219010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.224066] ** ** [ 0.229146] ** This means that this kernel is built to expose internal ** [ 0.238016] ** IOMMU data structures, which may compromise security on ** [ 0.241013] ** your system. ** [ 0.247017] ** ** [ 0.250011] ** If you see this message and you are not debugging the ** [ 0.252011] ** kernel, report this immediately to your vendor! ** [ 0.256015] ** ** [ 0.257073] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.260010] ************************************************************* [ 0.263085] NET: Registered protocol family 16 [ 0.265729] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.269069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.272076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.276126] cpuidle: using governor menu [ 0.277846] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.280646] PCI: Using configuration type 1 for base access [ 0.283150] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.289446] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.290000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.292181] cryptd: max_cpu_qlen set to 1000 [ 0.294309] ACPI: Added _OSI(Module Device) [ 0.295017] ACPI: Added _OSI(Processor Device) [ 0.296043] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.297009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.303262] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.312190] ACPI: Interpreter enabled [ 0.313101] ACPI: PM: (supports S0 S3 S4 S5) [ 0.315017] ACPI: Using IOAPIC for interrupt routing [ 0.317163] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.321684] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.335077] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.337044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.340017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.344136] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.349193] acpiphp: Slot [2] registered [ 0.351119] acpiphp: Slot [5] registered [ 0.352220] acpiphp: Slot [6] registered [ 0.354101] acpiphp: Slot [3] registered [ 0.356143] acpiphp: Slot [4] registered [ 0.357090] acpiphp: Slot [7] registered [ 0.359091] acpiphp: Slot [8] registered [ 0.360178] acpiphp: Slot [9] registered [ 0.362111] acpiphp: Slot [10] registered [ 0.364117] acpiphp: Slot [11] registered [ 0.366104] acpiphp: Slot [12] registered [ 0.367077] acpiphp: Slot [13] registered [ 0.369077] acpiphp: Slot [14] registered [ 0.370153] acpiphp: Slot [15] registered [ 0.372101] acpiphp: Slot [16] registered [ 0.374090] acpiphp: Slot [17] registered [ 0.375123] acpiphp: Slot [18] registered [ 0.378112] acpiphp: Slot [19] registered [ 0.379194] acpiphp: Slot [20] registered [ 0.381071] acpiphp: Slot [21] registered [ 0.383106] acpiphp: Slot [22] registered [ 0.384251] acpiphp: Slot [23] registered [ 0.386135] acpiphp: Slot [24] registered [ 0.387203] acpiphp: Slot [25] registered [ 0.389105] acpiphp: Slot [26] registered [ 0.391084] acpiphp: Slot [27] registered [ 0.392094] acpiphp: Slot [28] registered [ 0.394101] acpiphp: Slot [29] registered [ 0.395091] acpiphp: Slot [30] registered [ 0.397073] acpiphp: Slot [31] registered [ 0.398086] PCI host bridge to bus 0000:00 [ 0.400026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.402023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.404154] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.408027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.410024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.413028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.416180] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.419251] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.423451] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.432017] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.436054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.438019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.441017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.444020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.447441] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.450888] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.454037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.457999] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.463014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.473827] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.478021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.484703] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.491016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.498018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.513017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.524950] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.530017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.538017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.555019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.568000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.570360] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.573519] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.577449] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.580214] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.586151] iommu: Default domain type: Passthrough [ 0.588719] SCSI subsystem initialized [ 0.590165] ACPI: bus type USB registered [ 0.592101] usbcore: registered new interface driver usbfs [ 0.593060] usbcore: registered new interface driver hub [ 0.595144] usbcore: registered new device driver usb [ 0.597204] pps_core: LinuxPPS API ver. 1 registered [ 0.599012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.603075] PTP clock support registered [ 0.607070] EDAC MC: Ver: 3.0.0 [ 0.609173] PCI: Using ACPI for IRQ routing [ 0.612697] NetLabel: Initializing [ 0.614012] NetLabel: domain hash size = 128 [ 0.617010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.619137] NetLabel: unlabeled traffic allowed by default [ 0.622556] vgaarb: loaded [ 0.624324] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.626014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.636571] clocksource: Switched to clocksource kvm-clock [ 0.813711] VFS: Disk quotas dquot_6.6.0 [ 0.815575] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.818930] *** VALIDATE ramfs *** [ 0.820626] *** VALIDATE hugetlbfs *** [ 0.834388] pnp: PnP ACPI init [ 0.845160] pnp: PnP ACPI: found 6 devices [ 0.876054] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.880828] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.886631] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.892672] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.895992] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.900144] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.904351] NET: Registered protocol family 2 [ 0.911142] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.921411] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.929283] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.993800] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.005827] TCP: Hash tables configured (established 65536 bind 65536) [ 1.015872] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.026760] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.033899] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.046033] NET: Registered protocol family 1 [ 1.056075] RPC: Registered named UNIX socket transport module. [ 1.065070] RPC: Registered udp transport module. [ 1.072169] RPC: Registered tcp transport module. [ 1.076019] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.081328] NET: Registered protocol family 44 [ 1.087545] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.096861] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.102124] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.113576] pci 0000:00:01.0: quirk_isa_dma_hangs+0x0/0x20 took 11183 usecs [ 1.119630] PCI: CLS 0 bytes, default 64 [ 1.122822] Unpacking initramfs... [ 3.634570] debug: unmapping init [mem 0xffff8b903cc64000-0xffff8b903ffcffff] [ 3.641318] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.643817] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.648321] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.537982] Initialise system trusted keyrings [ 4.541170] Key type blacklist registered [ 4.544755] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.555509] zbud: loaded [ 4.559276] *** VALIDATE nfs *** [ 4.588652] *** VALIDATE nfs4 *** [ 4.625880] pstore: using deflate compression [ 4.661384] Platform Keyring initialized [ 4.773916] NET: Registered protocol family 38 [ 4.777768] Key type asymmetric registered [ 4.779914] Asymmetric key parser 'x509' registered [ 4.782332] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.785903] io scheduler mq-deadline registered [ 4.787966] io scheduler kyber registered [ 4.790384] io scheduler bfq registered [ 4.792562] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.796755] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.800736] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.804227] ACPI: Power Button [PWRF] [ 4.812253] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.820627] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.841067] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.899948] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.987040] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.075329] Non-volatile memory driver v1.3 [ 5.076898] Linux agpgart interface v0.103 [ 5.108346] virtio_blk virtio1: [vda] 145800 512-byte logical blocks (74.6 MB/71.2 MiB) [ 5.111550] vda: detected capacity change from 0 to 74649600 [ 5.125986] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.129376] vdb: detected capacity change from 0 to 1073741824 [ 5.135853] libphy: Fixed MDIO Bus: probed [ 5.140738] usbcore: registered new interface driver usbserial_generic [ 5.143779] usbserial: USB Serial support registered for generic [ 5.146315] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.150576] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.152542] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.155162] mousedev: PS/2 mouse device common for all mice [ 5.167992] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.173839] rtc_cmos 00:05: RTC can wake from S4 [ 5.178534] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.179524] rtc_cmos 00:05: registered as rtc0 [ 5.184904] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.188810] intel_pstate: CPU model not supported [ 5.209998] hpet1: lost 1 rtc interrupts [ 5.212535] hid: raw HID events driver (C) Jiri Kosina [ 5.214486] usbcore: registered new interface driver usbhid [ 5.216677] usbhid: USB HID core driver [ 5.218241] drop_monitor: Initializing network drop monitor service [ 5.218357] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.221037] Initializing XFRM netlink socket [ 5.226049] NET: Registered protocol family 10 [ 5.229691] Segment Routing with IPv6 [ 5.231778] NET: Registered protocol family 17 [ 5.234035] mpls_gso: MPLS GSO support [ 5.238503] RAS: Correctable Errors collector initialized. [ 5.240457] AVX version of gcm_enc/dec engaged. [ 5.242403] AES CTR mode by8 optimization enabled [ 5.323596] sched_clock: Marking stable (5323522504, 0)->(6208729692, -885207188) [ 5.327875] registered taskstats version 1 [ 5.332160] Loading compiled-in X.509 certificates [ 5.335618] zswap: loaded using pool lzo/zbud [ 5.385677] Key type big_key registered [ 5.408214] Key type encrypted registered [ 5.410335] ima: No TPM chip found, activating TPM-bypass! [ 5.414740] ima: Allocated hash algorithm: sha1 [ 5.418177] ima: No architecture policies found [ 5.421509] evm: Initialising EVM extended attributes: [ 5.426134] evm: security.selinux [ 5.427620] evm: security.ima [ 5.429376] evm: security.capability [ 5.432280] evm: HMAC attrs: 0x1 [ 5.437541] rtc_cmos 00:05: setting system clock to 2026-07-15 05:20:38 UTC (1784092838) [ 5.452884] debug: unmapping init [mem 0xffffffff8de03000-0xffffffff8dffffff] [ 5.460532] debug: unmapping init [mem 0xffffffff8cb82000-0xffffffff8ce58fff] [ 5.472171] Write protecting the kernel read-only data: 28672k [ 5.482580] debug: unmapping init [mem 0xffffffff8b203000-0xffffffff8b3fffff] [ 5.485026] debug: unmapping init [mem 0xffffffff8bb14000-0xffffffff8bbfffff] [ 5.548567] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.576558] systemd[1]: Detected virtualization kvm. [ 5.584384] systemd[1]: Detected architecture x86-64. [ 5.588877] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.645374] systemd[1]: No hostname configured. [ 5.650792] systemd[1]: Set hostname to . [ 5.653799] random: systemd: uninitialized urandom read (16 bytes read) [ 5.659171] systemd[1]: Initializing machine ID from random generator. [ 5.719526] random: ln: uninitialized urandom read (6 bytes read) [ 5.921536] random: systemd: uninitialized urandom read (16 bytes read) [ 5.924393] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.929474] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.935626] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.248399] device-mapper: uevent: version 1.0.3 [ 7.250376] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.302479] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 8.481227] scsi host0: ata_piix [ 8.517187] scsi host1: ata_piix [ 8.518781] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 8.523871] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 12.946256] random: crng init done [ 12.949368] random: 7 urandom warning(s) missed due to ratelimiting [ 15.034741] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.334453] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.559716] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.350046] SELinux: Disabled at runtime. [ 19.522526] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.547905] systemd[1]: Detected virtualization kvm. [ 19.556237] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.340332] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.352096] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.366595] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.376306] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.384674] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.408317] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.428403] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ 21.757565] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 22.471063] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.314717] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.375301] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.652761] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.703734] EDAC sbridge: Ver: 1.1.2 [ 27.029681] Key type dns_resolver registered [ 27.767384] NFS: Registering the id_resolver key type [ 27.769272] Key type id_resolver registered [ 27.770929] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg312-client login: [ 64.730577] libcfs: loading out-of-tree module taints kernel. [ 64.823960] Key type ._llcrypt registered [ 64.834267] Key type .llcrypt registered [ 65.112685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 65.119875] alg: No test for adler32 (adler32-zlib) [ 66.165975] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 66.553223] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 68.199210] Key type lgssc registered [ 69.094293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 128.940771] Lustre: Mounted lustre-client [ 131.549414] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 138.082583] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing check_logdir /tmp/testlogs/ [ 140.053713] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing yml_node [ 141.914938] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 142.976117] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 144.011090] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 144.627708] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Jul 15 01:22:57 EDT 2026 [ 150.304785] Lustre: Unmounted lustre-client [ 162.834195] Key type lgssc unregistered [ 162.968949] LNet: 8375:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 162.973227] LNetError: 8375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 162.986589] LNet: Removed LNI 192.168.203.12@tcp [ 163.306194] Key type .llcrypt unregistered [ 163.308779] Key type ._llcrypt unregistered [ 173.330792] Key type ._llcrypt registered [ 173.332993] Key type .llcrypt registered [ 173.426341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 173.435698] alg: No test for adler32 (adler32-zlib) [ 174.251472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 174.293068] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 174.305108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 175.970829] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 176.002484] LNet: 9248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 176.005153] LNetError: 9248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 176.014606] LNet: Removed LNI 192.168.203.12@tcp [ 176.294433] Key type .llcrypt unregistered [ 176.295853] Key type ._llcrypt unregistered [ 178.590604] Key type ._llcrypt registered [ 178.592444] Key type .llcrypt registered [ 179.159714] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 01:23:31 (1784093011) [ 179.300952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 179.307046] alg: No test for adler32 (adler32-zlib) [ 180.127779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 180.161670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 180.173321] LNet: 9666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 180.182311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 180.206636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 180.219737] LNet: 9673:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 183.088665] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 01:23:35 (1784093015) [ 183.160223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 183.172514] LNet: 10249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 183.183728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 183.214518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 185.942654] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 01:23:38 (1784093018) [ 185.998104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 186.016165] LNet: 10833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 186.028077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 186.064760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 188.714785] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 01:23:41 (1784093021) [ 188.767273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 188.783418] LNet: 11411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 188.798934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 188.835942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 189.164008] hrtimer: interrupt took 3979738 ns [ 191.631751] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 01:23:44 (1784093024) [ 191.669667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 191.695960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 191.741569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 191.948327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 194.879273] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 01:23:47 (1784093027) [ 194.917760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 194.937162] LNet: 12584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 194.942242] LNet: 12584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 194.960072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 195.010322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 195.199335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 195.387465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 198.226569] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 01:23:50 (1784093030) [ 198.264409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 198.285099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 198.323105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 201.166526] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 01:23:53 (1784093033) [ 201.208338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 201.240485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 201.283575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 201.306865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 201.504905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 201.526464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 201.715679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 201.737359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 201.917205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 201.945132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 202.145924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 202.167561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 202.357203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 202.378898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 202.570838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 202.591754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 202.759380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 202.775701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 202.949632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 202.967683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 203.129194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 203.144117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 203.302847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 203.321741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 203.507154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 203.528792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 203.710679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 203.731917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 206.842675] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 01:23:59 (1784093039) [ 206.879537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 206.892670] LNet: 14442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 206.896490] LNet: 14442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 206.909768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 206.956620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 206.979385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 210.075617] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 01:24:02 (1784093042) [ 210.118034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 210.137543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 210.180614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 210.207583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 213.316864] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 01:24:05 (1784093045) [ 213.362871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 213.381220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 213.430766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 213.450884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 216.511987] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 01:24:08 (1784093048) [ 216.549683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 216.579067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 216.628943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 216.646125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 219.524807] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 01:24:11 (1784093051) [ 219.571500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 219.594584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 219.638579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 219.662628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 222.614218] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 01:24:15 (1784093055) [ 222.649738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.675778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.724065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 222.746756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 225.932466] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 01:24:18 (1784093058) [ 225.977927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 225.990931] LNet: 17959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 225.995357] LNet: 17959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 226.011458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 226.073263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 226.273683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 229.313557] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 01:24:21 (1784093061) [ 229.370469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.397960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.505176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 229.617937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 232.678282] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 01:24:25 (1784093065) [ 232.760233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 232.790736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 232.846267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 232.954577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 235.923719] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 01:24:28 (1784093068) [ 235.969867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.992191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 236.251793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 236.377455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 239.754389] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 01:24:32 (1784093072) [ 239.810396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 239.841807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 239.903976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 239.934701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 243.119659] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 01:24:35 (1784093075) [ 243.164062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.197138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.236937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 243.265092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 246.350160] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 01:24:38 (1784093078) [ 246.411815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 246.433492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 246.476791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 249.430798] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 01:24:41 (1784093081) [ 249.466789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.496547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.547241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 252.482228] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 01:24:44 (1784093084) [ 252.521436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 252.547721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 252.608713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 252.822921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 255.844490] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 01:24:48 (1784093088) [ 255.892658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 255.917218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 255.973469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 255.993976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 259.015967] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 01:24:51 (1784093091) [ 259.060583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 259.076768] LNet: 23978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 259.080409] LNet: 23978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 259.099181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 259.150505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 259.176348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 262.065475] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 01:24:54 (1784093094) [ 262.095754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 262.115665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 262.155277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 262.176330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 265.033586] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:24:57 (1784093097) [ 265.076996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 265.100749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.145093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 265.172032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 267.964233] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:25:00 (1784093100) [ 268.010949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.035215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.062800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 268.100302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 268.111598] LNetError: 25754:0:(peer.c:2276:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 268.128542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 268.143801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 270.497783] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:25:03 (1784093103) [ 270.533281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 270.552539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 270.594651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.RvTNHr0w6Y [ 270.625333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.RvTNHr0w6Y [ 270.651410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.RvTNHr0w6Y [ 272.999841] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:25:05 (1784093105) [ 273.036462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.055985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.083506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 273.101254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 273.125103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 275.497557] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 01:25:08 (1784093108) [ 275.542211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 275.560824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 275.593934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 275.606340] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 276.277289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 276.817816] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 278.503182] Key type lgssc registered [ 278.941305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 280.370085] LNetError: 27513:0:(lib-move.c:2120:lnet_handle_find_routed_path()) peer 192.168.203.12@tcp99 has no available nets [ 281.529708] Key type lgssc unregistered [ 281.680919] LNetError: 28619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 281.693503] LNet: Removed LNI 192.168.203.12@tcp [ 281.934948] Key type .llcrypt unregistered [ 281.936595] Key type ._llcrypt unregistered [ 284.181728] Key type ._llcrypt registered [ 284.183015] Key type .llcrypt registered [ 284.706174] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:25:17 (1784093117) [ 284.836421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 284.843211] alg: No test for adler32 (adler32-zlib) [ 285.659152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 285.689345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 286.763605] LNet: Message forwarding will be enabled in 2s [ 289.823161] LNet: Message forwarding enabled [ 289.959932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 289.972146] LNet: Message forwarding will stop in 2s to 6s [ 294.155058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 294.652188] LNet: Message forwarding disabled [ 295.257526] LNet: Message forwarding will be enabled in 2s [ 297.863334] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:25:30 (1784093130) [ 297.908864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.921133] LNet: 30022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 298.539320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.574627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 300.887476] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:25:33 (1784093133) [ 300.921984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.933718] LNet: 30599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.945063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.973912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 300.988947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 301.004262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 301.020401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 301.041440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 301.060874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 301.076691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 301.094167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 301.112507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 301.128083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 301.144058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 301.159013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 301.175596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 301.192312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 301.207386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 301.223862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 301.242789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 301.260383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 301.278187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 301.288460] LNetError: 30649:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 303.753586] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:25:36 (1784093136) [ 303.793313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.802962] LNet: 31213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 303.814951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.868873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 306.568611] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:25:39 (1784093139) [ 307.353761] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg312-server [ 308.085276] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:25:40 (1784093140) [ 308.116386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.128908] LNet: 32144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 308.141712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.172866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 308.185901] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 308.556031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[13-17]@tcp [ 308.569274] LNet: 32282:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 311.296415] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:25:43 (1784093143) [ 311.647407] LNetError: 32157:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.14@tcp has gone from up to down [ 312.076133] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg312-server [ 312.758775] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:25:45 (1784093145) [ 312.793853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 314.847454] LNet: 33210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 314.850941] LNetError: 33210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 314.858642] LNet: Removed LNI 192.168.203.12@tcp [ 314.868191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.889082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 314.898466] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 315.162910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[13-18/2]@tcp [ 315.175898] LNet: 33335:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 315.181390] LNet: 33335:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 315.191385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[13-18/2]@tcp [ 317.582900] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:25:50 (1784093150) [ 317.620741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.687758] LNetError: 33922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 318.696478] LNet: Removed LNI 192.168.203.12@tcp [ 318.705542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.744347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 318.769921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.786811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.814219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 318.841029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.857636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.879824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 318.911481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.927129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.955820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 318.989338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.007959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.035664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 319.069683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.086545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.115409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 319.151618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.168559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.195386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 319.229320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.246662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.275130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 319.307866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.328550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.355137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 319.388792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.406186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.444926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 319.482309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.501922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.528345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 319.553325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.569701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.600882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 322.050808] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:25:54 (1784093154) [ 322.743297] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg312-server [ 323.385829] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:25:55 (1784093155) [ 324.163525] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg312-server [ 324.817150] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:25:57 (1784093157) [ 324.859788] LNet: 35320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.862912] LNet: 35320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 325.137291] Key type .llcrypt unregistered [ 325.139139] Key type ._llcrypt unregistered [ 327.258428] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 327.325340] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 327.327661] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 327.330610] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 329.536338] Key type ._llcrypt registered [ 329.537738] Key type .llcrypt registered [ 329.611619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 329.618874] alg: No test for adler32 (adler32-zlib) [ 330.450204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.479900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 330.495096] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 330.787309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 330.796576] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 331.630689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 331.641814] LNet: Removed LNI 10.1.2.3@tcp [ 331.695078] LNet: 36048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.698274] LNetError: 36048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 331.708506] LNet: Removed LNI 192.168.203.12@tcp [ 331.978478] Key type .llcrypt unregistered [ 331.979863] Key type ._llcrypt unregistered [ 336.106104] Key type ._llcrypt registered [ 336.107744] Key type .llcrypt registered [ 336.630553] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:26:09 (1784093169) [ 336.816478] Key type .llcrypt unregistered [ 336.818553] Key type ._llcrypt unregistered [ 339.007562] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 339.047422] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 339.050762] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 339.052971] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 341.291485] Key type ._llcrypt registered [ 341.293479] Key type .llcrypt registered [ 341.377137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 341.387586] alg: No test for adler32 (adler32-zlib) [ 342.207874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.264845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 342.280652] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 342.668415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 342.684446] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 343.949695] LNet: 37540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.954143] LNetError: 37540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 343.959397] LNet: Removed LNI 10.1.2.3@tcp [ 343.964870] LNet: Removed LNI 192.168.203.12@tcp [ 344.267449] Key type .llcrypt unregistered [ 344.270416] Key type ._llcrypt unregistered [ 348.751392] Key type ._llcrypt registered [ 348.753344] Key type .llcrypt registered [ 349.371929] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:26:21 (1784093181) [ 349.586311] Key type .llcrypt unregistered [ 349.588281] Key type ._llcrypt unregistered [ 351.933971] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 351.997894] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 352.000254] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 352.002819] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 354.326960] Key type ._llcrypt registered [ 354.331906] Key type .llcrypt registered [ 354.429373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 354.438658] alg: No test for adler32 (adler32-zlib) [ 355.264585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.290207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 355.303086] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 356.276490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 356.289490] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 357.404520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 357.417117] LNet: Removed LNI 10.1.2.3@tcp [ 357.429063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 357.453734] LNet: Removed LNI 10.1.2.31@tcp [ 357.465151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 357.482269] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 357.923511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 357.934920] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 358.437767] LNet: 39460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.441049] LNetError: 39460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 358.444834] LNet: Removed LNI 10.1.2.31@tcp [ 358.451597] LNet: Removed LNI 10.1.2.3@tcp [ 358.808445] Key type .llcrypt unregistered [ 358.809854] Key type ._llcrypt unregistered [ 364.452734] Key type ._llcrypt registered [ 364.455242] Key type .llcrypt registered [ 365.227878] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:26:37 (1784093197) [ 365.432085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 365.440295] alg: No test for adler32 (adler32-zlib) [ 366.292832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.322990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 366.336103] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 366.730605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.746780] LNet: 40411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.752190] LNetError: 40411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 366.764630] LNet: Removed LNI 192.168.203.12@tcp [ 366.784080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.819393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.12@tcp [ 366.841843] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 367.244919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.12@tcp [ 367.270241] LNet: Removed LNI 192.168.203.12@tcp [ 367.300125] LNet: 40546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.714391] Key type .llcrypt unregistered [ 367.717104] Key type ._llcrypt unregistered [ 373.060622] Key type ._llcrypt registered [ 373.062165] Key type .llcrypt registered [ 373.683503] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:26:46 (1784093206) [ 374.415508] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg312-server [ 375.090864] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:26:47 (1784093207) [ 375.277451] Key type .llcrypt unregistered [ 375.279435] Key type ._llcrypt unregistered [ 377.781385] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 377.890158] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 377.895967] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 377.898196] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 380.136755] Key type ._llcrypt registered [ 380.138688] Key type .llcrypt registered [ 380.227200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 380.235704] alg: No test for adler32 (adler32-zlib) [ 381.060629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.101468] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 381.463256] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 382.555460] LNet: 42381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.557943] LNetError: 42381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 382.560839] LNet: Removed LNI 192.168.203.12@tcp1 [ 382.570505] LNet: Removed LNI 10.1.2.3@tcp2 [ 382.876117] Key type .llcrypt unregistered [ 382.877994] Key type ._llcrypt unregistered [ 387.798282] Key type ._llcrypt registered [ 387.800091] Key type .llcrypt registered [ 388.387137] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:27:00 (1784093220) [ 388.539292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 388.549369] alg: No test for adler32 (adler32-zlib) [ 389.373653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.400263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 389.410752] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 389.758949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.770661] LNet: 43348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.774377] LNetError: 43348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 389.785578] LNet: Removed LNI 192.168.203.12@tcp [ 389.796910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.831778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 389.846951] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 392.583401] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:27:05 (1784093225) [ 392.616240] LNet: 43930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.619061] LNetError: 43930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 392.626505] LNet: Removed LNI 192.168.203.12@tcp [ 392.908637] Key type .llcrypt unregistered [ 392.910316] Key type ._llcrypt unregistered [ 395.160417] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 395.239645] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 395.246638] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 395.248915] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 397.493615] Key type ._llcrypt registered [ 397.495286] Key type .llcrypt registered [ 397.581991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 397.590260] alg: No test for adler32 (adler32-zlib) [ 398.422861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.454696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 398.474927] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 398.849533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 398.859665] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 399.962672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.972610] LNet: 44681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 399.975555] LNetError: 44681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 399.980304] LNet: Removed LNI 10.1.2.3@tcp [ 399.987898] LNet: Removed LNI 192.168.203.12@tcp [ 399.997577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.040664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 400.062735] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 400.071203] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 400.293508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.307720] LNet: Removed LNI 10.1.2.3@tcp [ 400.322645] LNet: Removed LNI 192.168.203.12@tcp [ 400.334375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.370666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 400.387160] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 400.390312] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 400.614087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 400.628192] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 401.132653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.146751] LNet: 44870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.154066] LNet: 44870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 401.162539] LNetError: 44870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 401.166066] LNetError: 44870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 401.169063] LNet: Removed LNI 10.1.2.3@tcp [ 401.172615] LNet: Removed LNI 192.168.203.12@tcp [ 401.186067] LNet: Removed LNI 192.168.203.12@tcp2 [ 401.203155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.236843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 401.249857] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 401.252954] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 401.255970] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 404.714442] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:27:17 (1784093237) [ 404.757281] LNet: 45458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.766644] LNetError: 45458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 404.771327] LNetError: 45458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 404.776521] LNet: Removed LNI 10.1.2.3@tcp [ 404.778499] LNet: Removed LNI 192.168.203.12@tcp [ 404.787063] LNet: Removed LNI 192.168.203.12@tcp2 [ 405.099650] Key type .llcrypt unregistered [ 405.102843] Key type ._llcrypt unregistered [ 407.552149] Key type ._llcrypt registered [ 407.554458] Key type .llcrypt registered [ 407.641556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 407.650371] alg: No test for adler32 (adler32-zlib) [ 408.469384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.497722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 408.512432] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 408.887327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.899458] LNet: 45860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.905502] LNetError: 45860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 408.922081] LNet: Removed LNI 192.168.203.12@tcp [ 408.937680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.972990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 408.988348] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 409.200431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.223219] LNet: Removed LNI 192.168.203.12@tcp [ 409.233459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.265247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 409.279341] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 409.508023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.520289] LNet: 45920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.524086] LNet: 45920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 409.527708] LNetError: 45920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 409.531018] LNetError: 45920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 409.541170] LNet: Removed LNI 192.168.203.12@tcp [ 409.553391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.586754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 409.605395] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 409.898868] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 409.980503] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 409.982738] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 409.990555] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 412.173441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.185172] LNet: 45998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.189551] LNetError: 45998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 412.207301] LNet: Removed LNI 192.168.203.12@tcp [ 412.220655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.256370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 412.277757] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 415.214371] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:27:27 (1784093247) [ 415.260666] LNet: 46586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 415.266027] LNetError: 46586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 415.277689] LNet: Removed LNI 192.168.203.12@tcp [ 415.546204] Key type .llcrypt unregistered [ 415.547425] Key type ._llcrypt unregistered [ 417.815700] Key type ._llcrypt registered [ 417.817308] Key type .llcrypt registered [ 417.902372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 417.909220] alg: No test for adler32 (adler32-zlib) [ 418.734733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.761834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 418.772705] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 419.114772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.122904] LNet: 46992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 419.126230] LNetError: 46992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 419.133363] LNet: Removed LNI 192.168.203.12@tcp [ 419.142206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.170110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 419.182105] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 421.600491] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:27:34 (1784093254) [ 421.646571] LNet: 47579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.649790] LNetError: 47579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 421.656616] LNet: Removed LNI 192.168.203.12@tcp [ 421.897471] Key type .llcrypt unregistered [ 421.899305] Key type ._llcrypt unregistered [ 424.099732] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 424.158183] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 424.160827] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 424.163740] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 426.375725] Key type ._llcrypt registered [ 426.377159] Key type .llcrypt registered [ 426.447881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 426.453746] alg: No test for adler32 (adler32-zlib) [ 427.277594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.310674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 427.322442] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 427.681635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.693343] LNet: 48033:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 427.697264] LNetError: 48033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 427.710401] LNet: Removed LNI 192.168.203.12@tcp [ 427.719517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.749494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 427.764788] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 430.336783] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:27:42 (1784093262) [ 430.412801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.421453] LNet: 48631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 430.424959] LNetError: 48631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 430.436093] LNet: Removed LNI 192.168.203.12@tcp [ 430.446545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.494068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 432.850690] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:27:45 (1784093265) [ 432.891157] LNet: 49208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 433.154521] Key type .llcrypt unregistered [ 433.156314] Key type ._llcrypt unregistered [ 435.375160] Key type ._llcrypt registered [ 435.376805] Key type .llcrypt registered [ 435.450893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 435.460701] alg: No test for adler32 (adler32-zlib) [ 436.268893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.294912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 436.306084] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 436.650112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.660828] LNet: 49610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 436.664576] LNetError: 49610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 436.683534] LNet: Removed LNI 192.168.203.12@tcp [ 436.697402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.726877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 436.740696] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 439.508883] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:27:52 (1784093272) [ 439.545738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.557462] LNet: 50197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 439.561971] LNetError: 50197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 439.576905] LNet: Removed LNI 192.168.203.12@tcp [ 439.592302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.623978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 442.021179] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:27:54 (1784093274) [ 442.621716] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 443.320991] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:27:55 (1784093275) [ 443.950813] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 444.598600] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:27:57 (1784093277) [ 445.203159] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 445.812503] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:27:58 (1784093278) [ 446.410227] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 447.034073] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:27:59 (1784093279) [ 447.637287] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 448.339347] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 01:28:00 (1784093280) [ 448.379743] LNet: 52244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 448.677881] Key type .llcrypt unregistered [ 448.679602] Key type ._llcrypt unregistered [ 451.003801] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 451.046653] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 451.049282] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 451.054048] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 453.278221] Key type ._llcrypt registered [ 453.279766] Key type .llcrypt registered [ 453.359567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 453.365354] alg: No test for adler32 (adler32-zlib) [ 454.189775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.229436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 454.240820] LNet: Added LNI 192.168.203.12@tcp [8/512/0/180] [ 454.244659] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 454.246566] LNet: 52572:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 454.294636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.306868] LNet: 52591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 454.311183] LNetError: 52591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 454.315374] LNet: Removed LNI 10.1.2.3@tcp [ 454.317243] LNetError: 52591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 454.328651] LNet: Removed LNI 192.168.203.12@tcp [ 454.342529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.384224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 454.404441] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 454.440755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.454930] LNet: 52615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 454.458489] LNetError: 52615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 454.474555] LNet: Removed LNI 192.168.203.12@tcp [ 454.487755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.527454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 454.542750] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 454.546516] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 454.548775] LNet: 52623:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 454.598878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.613457] LNet: 52641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 454.617316] LNetError: 52641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 454.621661] LNet: Removed LNI 10.1.2.3@tcp [ 454.623470] LNetError: 52641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 454.635305] LNet: Removed LNI 192.168.203.12@tcp [ 454.649733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.687952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 454.702653] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 454.706200] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 454.708266] LNet: 52649:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 454.779813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.792202] LNet: 52675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 454.796920] LNetError: 52675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 454.800068] LNet: Removed LNI 10.1.2.3@tcp [ 454.801713] LNetError: 52675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 454.810284] LNet: Removed LNI 192.168.203.12@tcp [ 454.821724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.854380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 454.867107] LNet: Added LNI 192.168.203.12@tcp [8/256/0/182] [ 454.869804] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 454.915522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.926621] LNet: 52703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 454.930542] LNetError: 52703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 454.934645] LNet: Removed LNI 10.1.2.3@tcp [ 454.936057] LNetError: 52703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 454.948336] LNet: Removed LNI 192.168.203.12@tcp [ 454.960425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.992920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 455.005768] LNet: Added LNI 192.168.203.12@tcp [8/256/0/182] [ 455.009137] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 455.010817] LNet: 52711:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 455.049139] LNet: Removed LNI 10.1.2.3@tcp [ 455.060795] LNet: Removed LNI 192.168.203.12@tcp [ 455.335764] Key type .llcrypt unregistered [ 455.337412] Key type ._llcrypt unregistered [ 459.557807] Key type ._llcrypt registered [ 459.559369] Key type .llcrypt registered [ 460.138635] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:28:12 (1784093292) [ 460.324272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 460.331905] alg: No test for adler32 (adler32-zlib) [ 461.154791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.181969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 461.192974] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 461.550627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.12 [ 461.561630] LNet: 53685:0:(config.c:1739:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 461.568923] LNet: Removed LNI 192.168.203.12@tcp [ 464.260198] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:28:16 (1784093296) [ 464.296152] LNet: 54246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.299702] LNetError: 54246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 464.309657] LNet: Removed LNI 192.168.203.12@tcp [ 464.593530] Key type .llcrypt unregistered [ 464.595356] Key type ._llcrypt unregistered [ 466.951488] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 467.065417] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 469.379136] Key type ._llcrypt registered [ 469.381074] Key type .llcrypt registered [ 469.453287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 469.460451] alg: No test for adler32 (adler32-zlib) [ 470.445779] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 472.952811] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:28:25 (1784093305) [ 473.003596] LNet: 55159:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 473.007489] LNetError: 55159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 473.017420] LNet: Removed LNI 10.1.2.3@tcp [ 473.285265] Key type .llcrypt unregistered [ 473.287239] Key type ._llcrypt unregistered [ 475.631273] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 475.753194] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 475.755832] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 475.758962] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 478.097228] Key type ._llcrypt registered [ 478.098621] Key type .llcrypt registered [ 478.170742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 478.179095] alg: No test for adler32 (adler32-zlib) [ 479.044183] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 481.887217] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:28:34 (1784093314) [ 482.534439] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 483.238332] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:28:35 (1784093315) [ 483.272765] LNet: 56341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 483.277358] LNetError: 56341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 483.285920] LNet: Removed LNI 10.1.2.3@tcp [ 483.583488] Key type .llcrypt unregistered [ 483.585096] Key type ._llcrypt unregistered [ 485.999910] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 486.114974] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 486.116584] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 486.119157] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 488.432626] Key type ._llcrypt registered [ 488.434012] Key type .llcrypt registered [ 488.518595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 488.531224] alg: No test for adler32 (adler32-zlib) [ 489.339389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.377846] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 493.050412] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:28:45 (1784093325) [ 493.237326] LNet: 57577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.240632] LNetError: 57577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 493.252590] LNet: Removed LNI 10.1.2.3@tcp [ 493.538540] Key type .llcrypt unregistered [ 493.540654] Key type ._llcrypt unregistered [ 495.952586] Key type ._llcrypt registered [ 495.954135] Key type .llcrypt registered [ 496.053886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 496.065618] alg: No test for adler32 (adler32-zlib) [ 496.884518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 496.900845] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 500.206407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 500.480809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_interrupt -r 1 [ 500.502482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_interrupt -r 1 [ 500.520521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 500.530762] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 500.611371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 500.624709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 500.638328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 500.653538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 500.668888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 500.684866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 500.698788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 500.713111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 500.722591] LNet: 58013:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 500.731713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 500.748300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 500.764220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 500.779420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 500.795479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 500.810435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 500.826062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 501.050269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_dropped -r 1 [ 501.068731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_dropped -r 1 [ 501.086684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 501.095776] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 501.185760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 501.201373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 501.216216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 501.230828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 501.245445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 501.261346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 501.275772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 501.293730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 501.304164] LNet: 58123:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 501.314097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 501.328265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 501.342540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 501.358331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 501.374679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 501.391687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 501.409620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 501.632987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_aborted -r 1 [ 501.654530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_aborted -r 1 [ 501.674582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 501.781240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 501.798465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 501.814988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 501.831552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 501.846483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 501.861351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 501.879166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 501.896499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 501.913637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 501.931963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 501.946504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 501.962622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 501.981048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 501.998272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 502.015345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 502.231810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_no_route -r 1 [ 502.248920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_no_route -r 1 [ 502.268595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 502.279485] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 502.283040] LNet: Skipped 1 previous similar message [ 502.383813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 502.400547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 502.415563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 502.431456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 502.446761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 502.460362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 502.476238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 502.490978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 502.500392] LNet: 58341:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 502.504063] LNet: 58341:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 502.513763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 502.531523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 502.547726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 502.565199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 502.583285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 502.599420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 502.615352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 502.837387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_timeout -r 1 [ 502.855624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_timeout -r 1 [ 502.874645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 502.977190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 502.994570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 503.010853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 503.024921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 503.040040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 503.053721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 503.069597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 503.073419] LNet: 1 local NIs in recovery (showing 1): 192.168.203.12@tcp [ 503.084942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 503.103711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 503.120373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 503.135073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 503.148965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 503.163384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 503.178446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 503.196761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 503.406432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_error -r 1 [ 503.424324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_error -r 1 [ 503.442451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 503.547780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 503.563365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 503.577649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 503.591962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 503.606927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 503.620515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 503.635823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 503.652749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 503.667859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 503.686257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 503.702239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 503.718819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 503.735469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 503.750454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 503.765443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 503.875429] LNet: 58600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 503.879045] LNetError: 58600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 503.886508] LNet: Removed LNI 192.168.203.12@tcp [ 504.048531] Key type .llcrypt unregistered [ 504.050560] Key type ._llcrypt unregistered [ 510.396943] Key type ._llcrypt registered [ 510.398639] Key type .llcrypt registered [ 510.930588] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:29:03 (1784093343) [ 511.261420] Key type .llcrypt unregistered [ 511.262944] Key type ._llcrypt unregistered [ 513.538402] Key type ._llcrypt registered [ 513.540133] Key type .llcrypt registered [ 513.614935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 513.622628] alg: No test for adler32 (adler32-zlib) [ 514.432830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 514.446547] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 517.533708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 521.422894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.439242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 521.453205] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 522.050694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_interrupt -r 1 [ 522.071090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_interrupt -r 1 [ 522.090344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_interrupt -r 1 [ 522.107909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_interrupt -r 1 [ 522.128613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_interrupt -r 1 [ 522.147915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_interrupt -r 1 [ 522.169388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_interrupt -r 1 [ 522.189774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_interrupt -r 1 [ 522.209777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 522.220501] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 522.223720] LNet: 1 local NIs in recovery (showing 1): 192.168.203.12@tcp [ 522.313660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 522.331977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 522.349532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 522.368744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 522.388469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 522.404335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 522.423499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 522.436960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 522.446723] LNet: 60063:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 522.456335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 522.472694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 522.491185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 522.508455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 522.526738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 522.549279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 522.568160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 522.803929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_dropped -r 1 [ 522.824882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_dropped -r 1 [ 522.843716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_dropped -r 1 [ 522.865880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_dropped -r 1 [ 522.887121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_dropped -r 1 [ 522.907361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_dropped -r 1 [ 522.928601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_dropped -r 1 [ 522.947347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_dropped -r 1 [ 522.966860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 522.976672] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 522.981251] LNet: Skipped 2 previous similar messages [ 523.090717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 523.109149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 523.131605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 523.149356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 523.164766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 523.185138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 523.202262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 523.218906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 523.228551] LNet: 60184:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 523.238452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 523.254342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 523.270863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 523.290104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 523.307050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 523.324540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 523.340982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 523.554741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_aborted -r 1 [ 523.573381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_aborted -r 1 [ 523.594028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_aborted -r 1 [ 523.614822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_aborted -r 1 [ 523.637399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_aborted -r 1 [ 523.661307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_aborted -r 1 [ 523.681353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_aborted -r 1 [ 523.700996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_aborted -r 1 [ 523.719864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 523.832511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 523.849373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 523.865326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 523.880840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 523.897943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 523.915183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 523.933379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 523.953821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 523.969769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 523.986486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 524.008338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 524.026155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 524.046130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 524.063311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 524.078595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 524.297817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_no_route -r 1 [ 524.318515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_no_route -r 1 [ 524.345093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_no_route -r 1 [ 524.364864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_no_route -r 1 [ 524.387806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_no_route -r 1 [ 524.409120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_no_route -r 1 [ 524.431479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_no_route -r 1 [ 524.449923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_no_route -r 1 [ 524.469713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 524.481967] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 524.485809] LNet: Skipped 6 previous similar messages [ 524.576987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 524.592153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 524.608961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 524.625267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 524.642991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 524.658410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 524.673862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 524.689654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 524.699290] LNet: 60426:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 524.703402] LNet: 60426:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 524.714202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 524.733171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 524.749969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 524.764788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 524.781438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 524.798085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 524.815916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 525.055422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_timeout -r 1 [ 525.073211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_timeout -r 1 [ 525.094261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_timeout -r 1 [ 525.113686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_timeout -r 1 [ 525.133872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_timeout -r 1 [ 525.153939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_timeout -r 1 [ 525.177465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_timeout -r 1 [ 525.197824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_timeout -r 1 [ 525.220539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 525.353447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 525.370561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 525.388969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 525.405728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 525.421300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 525.437782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 525.454311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 525.471465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 525.489465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 525.506189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 525.523723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 525.542374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 525.560517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 525.579665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 525.598963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 525.838073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_error -r 1 [ 525.858495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_error -r 1 [ 525.876055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_error -r 1 [ 525.897893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_error -r 1 [ 525.916148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_error -r 1 [ 525.935932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_error -r 1 [ 525.955384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_error -r 1 [ 525.975812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_error -r 1 [ 525.995768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 526.099646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 526.115180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 526.131208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 526.148021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 526.165854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 526.182446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 526.199398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 526.216622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 526.230585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 526.248434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 526.266121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 526.286890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 526.307898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 526.325403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 526.344448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 528.819858] LNet: 60745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 528.823866] LNetError: 60745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 528.829446] LNet: Removed LNI 192.168.203.12@tcp [ 529.897688] LNet: Removed LNI 192.168.203.12@tcp1 [ 530.090637] Key type .llcrypt unregistered [ 530.092307] Key type ._llcrypt unregistered [ 536.460531] Key type ._llcrypt registered [ 536.463087] Key type .llcrypt registered [ 537.031987] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:29:29 (1784093369) [ 537.372712] Key type .llcrypt unregistered [ 537.374635] Key type ._llcrypt unregistered [ 539.752860] Key type ._llcrypt registered [ 539.755205] Key type .llcrypt registered [ 539.854487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 539.863204] alg: No test for adler32 (adler32-zlib) [ 540.682196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 540.696623] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 544.004967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 544.296175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e remote_dropped -r 1 [ 544.315496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e remote_dropped -r 1 [ 544.335171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 544.346791] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 544.439313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 544.453774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 544.470617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 544.488184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 544.506096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 544.524379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 544.542321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 544.561796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 544.575089] LNet: 61991:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 544.591136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 544.613145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 544.636313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 544.657557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 544.678034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 544.695575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 544.713517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 544.977985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e remote_error -r 1 [ 544.998993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e remote_error -r 1 [ 545.019239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 545.030579] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 545.127464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 545.145962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 545.163606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 545.181060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 545.198703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 545.216482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 545.233781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 545.252667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 545.262661] LNet: 62100:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 545.276292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 545.293393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 545.312562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 545.329893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 545.345373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 545.367221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 545.390493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 545.650275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e remote_timeout -r 1 [ 545.671313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e remote_timeout -r 1 [ 545.697917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 545.804063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 545.817748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 545.836773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 545.857545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 545.876557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 545.894175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 545.912465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 545.931824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 545.948222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 545.969568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 545.988472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 546.005432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 546.023052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 546.040433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 546.060882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 546.189623] LNet: 62250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.192490] LNetError: 62250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 546.201769] LNet: Removed LNI 192.168.203.12@tcp [ 546.389482] Key type .llcrypt unregistered [ 546.391081] Key type ._llcrypt unregistered [ 552.832927] Key type ._llcrypt registered [ 552.834600] Key type .llcrypt registered [ 553.415198] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:29:45 (1784093385) [ 553.736603] Key type .llcrypt unregistered [ 553.738448] Key type ._llcrypt unregistered [ 556.117901] Key type ._llcrypt registered [ 556.119497] Key type .llcrypt registered [ 556.194371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 556.205239] alg: No test for adler32 (adler32-zlib) [ 557.022654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 557.036597] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 560.365157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 563.512204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.528844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 563.539867] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 564.067585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e remote_dropped -r 1 [ 564.085115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e remote_dropped -r 1 [ 564.104517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e remote_dropped -r 1 [ 564.121266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e remote_dropped -r 1 [ 564.139120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e remote_dropped -r 1 [ 564.157845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e remote_dropped -r 1 [ 564.176751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e remote_dropped -r 1 [ 564.196959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e remote_dropped -r 1 [ 564.215200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 564.226717] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 564.230732] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.112@tcp [ 564.331286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 564.348673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 564.363238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 564.378223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 564.394877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 564.410875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 564.432050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 564.447097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 564.457869] LNet: 63712:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 564.468098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 564.486308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 564.501661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 564.519618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 564.539183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 564.557267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 564.573654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 564.590315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 564.793984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e remote_error -r 1 [ 564.811350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e remote_error -r 1 [ 564.828702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e remote_error -r 1 [ 564.848938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e remote_error -r 1 [ 564.871735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e remote_error -r 1 [ 564.888207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e remote_error -r 1 [ 564.904906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e remote_error -r 1 [ 564.921989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e remote_error -r 1 [ 564.940719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 564.950957] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 564.954313] LNet: Skipped 2 previous similar messages [ 565.046921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 565.062315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 565.079420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 565.094212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 565.109856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 565.124853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 565.140487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 565.157001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 565.167646] LNet: 63835:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 565.178854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 565.195230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 565.209966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 565.224859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 565.239548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 565.255245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 565.271687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 565.289107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 565.519160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e remote_timeout -r 1 [ 565.551064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e remote_timeout -r 1 [ 565.576962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e remote_timeout -r 1 [ 565.600797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e remote_timeout -r 1 [ 565.619765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e remote_timeout -r 1 [ 565.640287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e remote_timeout -r 1 [ 565.662658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e remote_timeout -r 1 [ 565.683873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e remote_timeout -r 1 [ 565.702098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 565.812186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 565.829197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 565.845829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 565.862427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 565.878622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 565.895558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 565.914452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 565.929133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 565.943992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 565.962057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 565.979722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 565.998793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 566.013777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 566.031912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 566.051099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 566.069202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 568.511909] LNet: 64036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 568.515615] LNetError: 64036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 568.520767] LNet: Removed LNI 192.168.203.12@tcp [ 568.530278] LNet: Removed LNI 192.168.203.12@tcp1 [ 568.692580] Key type .llcrypt unregistered [ 568.694478] Key type ._llcrypt unregistered [ 574.944815] Key type ._llcrypt registered [ 574.946278] Key type .llcrypt registered [ 575.522456] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:30:07 (1784093407) [ 576.138656] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 576.884984] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:30:09 (1784093409) [ 577.275439] Key type .llcrypt unregistered [ 577.277441] Key type ._llcrypt unregistered [ 579.576703] Key type ._llcrypt registered [ 579.577793] Key type .llcrypt registered [ 579.647620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 579.654285] alg: No test for adler32 (adler32-zlib) [ 580.473542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 580.485971] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 583.993266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 584.290169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e network_timeout -r 1 [ 584.309138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e network_timeout -r 1 [ 584.330917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 584.340940] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 584.444526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 584.462293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 584.479802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 584.493801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 584.510399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 584.527054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 584.552658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 584.569831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 584.578718] LNet: 65576:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 584.588742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 584.607082] LNet: 1 local NIs in recovery (showing 1): 192.168.203.12@tcp [ 584.608265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 584.633560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 584.650936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 584.670480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 584.691496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 584.840545] LNet: 65615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.848503] LNetError: 65615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 584.861555] LNet: Removed LNI 192.168.203.12@tcp [ 585.030622] Key type .llcrypt unregistered [ 585.032332] Key type ._llcrypt unregistered [ 592.571291] Key type ._llcrypt registered [ 592.574825] Key type .llcrypt registered [ 592.654055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 592.661582] alg: No test for adler32 (adler32-zlib) [ 593.480496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 593.494223] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 596.750127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 599.923369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.940111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 599.950677] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 600.541338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e network_timeout -r 1 [ 600.557552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e network_timeout -r 1 [ 600.573580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e network_timeout -r 1 [ 600.592124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e network_timeout -r 1 [ 600.609764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e network_timeout -r 1 [ 600.627055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e network_timeout -r 1 [ 600.646494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e network_timeout -r 1 [ 600.664491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e network_timeout -r 1 [ 600.684835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 600.694914] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 600.799091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 600.818485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 600.835212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 600.850840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 600.867684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 600.885455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 600.902733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 600.919951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 600.931467] LNet: 66491:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 600.942161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 600.959510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 600.977361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 601.001894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 601.020451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 601.038110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 603.330343] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 603.475202] LNet: 66564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 603.480775] LNetError: 66564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 603.485626] LNet: Removed LNI 192.168.203.12@tcp [ 603.493734] LNet: Removed LNI 192.168.203.12@tcp1 [ 603.667535] Key type .llcrypt unregistered [ 603.668935] Key type ._llcrypt unregistered [ 610.152653] Key type ._llcrypt registered [ 610.153722] Key type .llcrypt registered [ 610.717431] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:30:43 (1784093443) [ 610.851789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 610.859645] alg: No test for adler32 (adler32-zlib) [ 611.676967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.702579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 611.714216] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 612.055652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 612.066525] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 612.454494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 612.490322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 612.515990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 612.536834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 612.558617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 612.578899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 612.599845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.12@tcp [ 612.611799] LNetError: 67662:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.203.12@tcp to recovery [ 612.703258] LNet: 1 local NIs in recovery (showing 1): 192.168.203.12@tcp [ 612.703304] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 614.751497] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 615.812623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 618.847375] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 619.022346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 619.040359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 619.065625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.075547] LNet: 67770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 619.078485] LNetError: 67770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 619.082177] LNet: Removed LNI 192.168.203.12@tcp [ 619.093436] LNet: Removed LNI 192.168.203.12@tcp1 [ 619.103205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.128107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 619.144298] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 619.452438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 619.461702] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 619.828826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 619.846791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 619.872135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 619.897185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 619.914387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 619.933974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 619.951350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 619.971701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.12@tcp [ 619.983860] LNetError: 68042:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.203.12@tcp to recovery [ 622.175272] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 622.178652] LNet: Skipped 1 previous similar message [ 623.186681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 626.271250] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 630.507146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 630.525405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 630.544509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 633.113601] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:31:05 (1784093465) [ 633.151495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.163567] LNet: 68754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.167516] LNetError: 68754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.171912] LNetError: 68754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 633.175765] LNet: Removed LNI 192.168.203.12@tcp [ 633.191896] LNet: Removed LNI 192.168.203.12@tcp1 [ 633.200935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.225768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 633.238824] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 633.556242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 633.568050] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 633.911113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 633.944378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 633.960391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 633.977573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 633.992727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 634.009645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 634.025894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.12@tcp --health 0 [ 634.071031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 634.271202] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.12@tcp [ 635.295329] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 635.298666] LNet: Skipped 1 previous similar message [ 650.593334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 650.643974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 650.657929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 650.673046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.12@tcp --health 500 [ 650.716015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 656.967987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 656.995730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.005870] LNet: 69266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 657.010407] LNetError: 69266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 657.013723] LNetError: 69266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 657.017504] LNet: Removed LNI 192.168.203.12@tcp [ 657.024328] LNet: Removed LNI 192.168.203.12@tcp1 [ 657.033451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.061145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 657.076700] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 657.415196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 657.426285] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 657.820703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 657.845357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 657.875284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 657.889967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 657.909427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 657.927173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 657.947396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 657.967992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.12@tcp --health 0 [ 658.080105] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 658.084719] LNet: Skipped 3 previous similar messages [ 659.099590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 668.503962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 668.520169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 668.535102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 671.124398] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 01:31:43 (1784093503) [ 671.333965] LNet: 70313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.337212] LNetError: 70313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.343210] LNetError: 70313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 671.349101] LNet: Removed LNI 192.168.203.12@tcp [ 671.362794] LNet: Removed LNI 192.168.203.12@tcp1 [ 671.632751] Key type .llcrypt unregistered [ 671.634540] Key type ._llcrypt unregistered [ 674.033581] Key type ._llcrypt registered [ 674.035196] Key type .llcrypt registered [ 674.122695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 674.130197] alg: No test for adler32 (adler32-zlib) [ 674.946329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 674.960893] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 678.883117] Lustre: DEBUG MARKER: Initial discovery [ 678.903503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.112@tcp [ 679.910951] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 679.933973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 679.957068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.112@tcp [ 679.968398] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 679.980853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 679.998409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.112@tcp -r 1 -m GET -l 3 [ 680.023599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.112@tcp -r 1 -m GET -e local_error [ 680.043094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.112@tcp -r 1 -m PUT -l 6 [ 680.095295] LNet: 1 local NIs in recovery (showing 1): 192.168.203.12@tcp [ 680.710677] Lustre: DEBUG MARKER: Force oleg312-client.virtnet to discover 192.168.203.112@tcp (in background) [ 680.734845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.112@tcp [ 681.323319] Lustre: DEBUG MARKER: Force oleg312-server to discover 192.168.203.12@tcp [ 682.292163] Lustre: DEBUG MARKER: Wait for 71066 [ 683.807557] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 684.421144] Lustre: DEBUG MARKER: Finished wait on 71066 [ 684.441216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 684.464939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 684.600759] LNet: 71530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.605124] LNetError: 71530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 684.613550] LNet: Removed LNI 192.168.203.12@tcp [ 684.794242] Key type .llcrypt unregistered [ 684.797053] Key type ._llcrypt unregistered [ 690.635819] Key type ._llcrypt registered [ 690.637231] Key type .llcrypt registered [ 691.108522] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 01:32:03 (1784093523) [ 691.286352] Key type .llcrypt unregistered [ 691.287816] Key type ._llcrypt unregistered [ 693.447304] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 693.492099] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 693.495180] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 693.498549] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 695.743552] Key type ._llcrypt registered [ 695.745404] Key type .llcrypt registered [ 695.819075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 695.826078] alg: No test for adler32 (adler32-zlib) [ 696.636731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.662038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 696.675468] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 696.983915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 696.993832] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 700.321855] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 01:32:12 (1784093532) [ 700.357548] LNet: 73596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.361874] LNetError: 73596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 700.365578] LNet: Removed LNI 10.1.2.3@tcp [ 700.376904] LNet: Removed LNI 192.168.203.12@tcp [ 700.653398] Key type .llcrypt unregistered [ 700.654964] Key type ._llcrypt unregistered [ 703.132731] Key type ._llcrypt registered [ 703.134327] Key type .llcrypt registered [ 703.212650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 703.220782] alg: No test for adler32 (adler32-zlib) [ 704.038107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.065142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 704.078101] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 704.399309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 704.409716] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 704.825855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.12@tcp [ 705.887207] LNetError: Unexpected error -22 connecting to 192.168.203.12@tcp at host 192.168.203.12:988 [ 707.316813] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 01:32:19 (1784093539) [ 707.348564] LNet: 74726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.352764] LNetError: 74726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 707.356669] LNet: Removed LNI 10.1.2.3@tcp [ 707.363755] LNet: Removed LNI 192.168.203.12@tcp [ 707.643518] Key type .llcrypt unregistered [ 707.645102] Key type ._llcrypt unregistered [ 709.826131] Key type ._llcrypt registered [ 709.827540] Key type .llcrypt registered [ 709.899631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 709.906207] alg: No test for adler32 (adler32-zlib) [ 710.718502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.742113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 710.753499] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 711.040698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 711.051632] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 711.393295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.12@tcp1 --nid 192.168.203.12@tcp2 [ 713.002449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp1 192.168.203.12@tcp2 [ 713.013308] LNetError: 75568:0:(lib-move.c:2063:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.12@tcp1 to 192.168.203.12@tcp2 - Route table may be misconfigured [ 713.023828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp2 192.168.203.12@tcp1 [ 715.385577] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 01:32:27 (1784093547) [ 715.430353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.442046] LNet: 76136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.446405] LNetError: 76136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 715.450921] LNet: Removed LNI 192.168.203.12@tcp1 [ 716.517925] LNet: Removed LNI 192.168.203.12@tcp2 [ 716.530175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.555631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 716.569043] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 716.889468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 716.899400] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 717.253173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 717.270643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp 192.168.203.12@tcp [ 717.289683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp1 192.168.203.12@tcp1 [ 717.308485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.12@tcp -d 192.168.203.12@tcp -e network_timeout [ 717.331060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -e network_timeout [ 717.350886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -e network_timeout [ 717.368072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -e network_timeout [ 717.388232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 717.397634] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 717.407618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 717.450235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 717.494736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 717.599086] LNet: 1 local NIs in recovery (showing 1): 192.168.203.12@tcp [ 719.714700] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 01:32:32 (1784093552) [ 719.750539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.761341] LNet: 76996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.764722] LNetError: 76996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 719.769299] LNetError: 76996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 719.773280] LNet: Removed LNI 192.168.203.12@tcp [ 720.805735] LNet: Removed LNI 192.168.203.12@tcp1 [ 720.814104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.843590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 720.936434] LNet: 77038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.080300] Key type .llcrypt unregistered [ 721.081625] Key type ._llcrypt unregistered [ 724.771912] Key type ._llcrypt registered [ 724.772932] Key type .llcrypt registered [ 725.246107] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 01:32:37 (1784093557) [ 725.364849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 725.371350] alg: No test for adler32 (adler32-zlib) [ 726.165643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.654259] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 727.262193] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 01:32:39 (1784093559) [ 727.299281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.309920] LNet: 78131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.321396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.348992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 727.360865] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 727.626862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 727.635845] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 727.989741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 728.009425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp1 [ 728.025614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp1 [ 730.352617] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 01:32:42 (1784093562) [ 731.022352] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg312-server [ 731.651061] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 01:32:44 (1784093564) [ 732.281123] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg312-server [ 732.824928] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 01:32:45 (1784093565) [ 733.427252] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg312-server [ 734.007325] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 01:32:46 (1784093566) [ 734.587903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg312-server [ 735.170323] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 01:32:47 (1784093567) [ 735.841786] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg312-server [ 736.454603] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 01:32:49 (1784093569) [ 737.110629] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg312-server [ 737.690325] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 01:32:50 (1784093570) [ 738.346738] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg312-server [ 738.910418] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 01:32:51 (1784093571) [ 739.548649] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg312-server [ 740.135754] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 01:32:52 (1784093572) [ 740.782072] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg312-server [ 741.375059] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 01:32:53 (1784093573) [ 741.409822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.421158] LNet: 82003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.424770] LNetError: 82003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.428356] LNet: Removed LNI 192.168.203.12@tcp [ 742.437603] LNet: Removed LNI 192.168.203.12@tcp1 [ 742.445745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.466919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 742.476518] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 742.727401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 742.757152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 742.821112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 742.829621] LNet: 82154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.832925] LNetError: 82154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.835566] LNetError: 82154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 742.849362] LNet: Removed LNI 192.168.203.12@tcp [ 742.858744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.881763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 742.892595] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 743.181340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 743.259492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 744.293271] LNet: Removed LNI 192.168.203.12@tcp [ 744.299750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.320683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 744.331949] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 744.600291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 744.631977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 744.696542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 744.706911] LNetError: 82446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 744.709875] LNetError: 82446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 744.718286] LNet: Removed LNI 192.168.203.12@tcp [ 744.726317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.746665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 744.757408] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 745.008576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 745.084822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.095425] LNet: 82591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.098479] LNet: 82591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 746.149442] LNet: Removed LNI 192.168.203.12@tcp [ 746.157243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.176546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 746.185269] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 746.424780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 746.453734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 746.522947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.557303] LNet: Removed LNI 192.168.203.12@tcp [ 747.566330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.588487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 747.599164] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 747.873991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 747.954580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.965680] LNetError: 82884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 747.968596] LNetError: 82884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 747.977548] LNet: Removed LNI 192.168.203.12@tcp [ 747.987061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.011250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 748.021960] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 748.292634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 748.326870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 748.410558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.477310] LNet: Removed LNI 192.168.203.12@tcp [ 749.486232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.509639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 749.520392] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 749.791499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 749.874953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.884420] LNet: 83176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.886800] LNet: 83176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 750.949338] LNet: Removed LNI 192.168.203.12@tcp [ 750.958478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.981565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 750.991121] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 751.239750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 751.269293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 751.358436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.421303] LNet: Removed LNI 192.168.203.12@tcp [ 752.429312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.452416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 752.462037] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 752.734551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 753.881160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.891547] LNetError: 83478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.895190] LNetError: 83478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 753.905371] LNet: Removed LNI 192.168.203.12@tcp [ 753.913407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.935072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 753.945131] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 754.213674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 754.251486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 754.335736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 755.365335] LNet: Removed LNI 192.168.203.12@tcp [ 755.375042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.398591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 755.409896] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 755.688028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 755.777496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 756.837450] LNet: Removed LNI 192.168.203.12@tcp [ 756.847443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.874338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 756.886377] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 757.138211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 757.162346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.176527] LNet: Removed LNI 192.168.203.12@tcp [ 757.184168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.204946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 757.215087] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 757.479225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 757.517563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.531355] LNet: Removed LNI 192.168.203.12@tcp [ 757.539435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.559216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 757.590528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.605636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.626798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 757.656586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.671555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.692637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 757.724193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.739485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.760906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 757.772140] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 758.038077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.046195] LNet: 84209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.048485] LNet: 84209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 758.059266] LNet: Removed LNI 192.168.203.12@tcp [ 758.067174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.094736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 760.420701] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 01:33:12 (1784093592) [ 760.456983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.476221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.509783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 760.523545] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/180] [ 760.978258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.993801] LNet: Removed LNI 192.168.203.12@tcp231 [ 761.001816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.030703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 761.041945] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/180] [ 761.044430] LNet: 84936:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 761.202194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 761.219640] LNet: Removed LNI 192.168.203.12@tcp231 [ 761.226624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 761.237303] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/180] [ 761.667633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.687350] LNet: Removed LNI 192.168.203.12@tcp231 [ 761.696462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.724166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 761.735056] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/0] [ 761.737708] LNet: 85088:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 761.885401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 761.901640] LNet: Removed LNI 192.168.203.12@tcp231 [ 761.909319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 761.919814] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/0] [ 762.378586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.392821] LNetError: 85231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.397193] LNetError: 85231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 762.410613] LNet: Removed LNI 192.168.203.12@tcp231 [ 762.420184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.450320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 762.461876] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/60] [ 762.625904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 762.642624] LNet: Removed LNI 192.168.203.12@tcp231 [ 762.651237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 762.662596] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/60] [ 763.090902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.107587] LNet: Removed LNI 192.168.203.12@tcp231 [ 763.116892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.154333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 763.167616] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/180] [ 763.170265] LNet: 85390:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 763.173277] LNet: 85390:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 763.328668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 763.346498] LNet: Removed LNI 192.168.203.12@tcp231 [ 763.352699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 763.363392] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/180] [ 765.872545] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 01:33:18 (1784093598) [ 765.905504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.921696] LNet: Removed LNI 192.168.203.12@tcp231 [ 765.932599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.960040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 765.972271] LNet: Added LNI 192.168.203.12@tcp232 [8/256/0/180] [ 766.235031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 766.251858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 766.267433] LNet: Removed LNI 192.168.203.12@tcp232 [ 766.276342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 766.287164] LNet: Added LNI 192.168.203.12@tcp232 [8/256/0/180] [ 766.558027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 766.573186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 766.591398] LNet: Removed LNI 192.168.203.12@tcp232 [ 766.599511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 766.608560] LNet: Added LNI 192.168.203.12@tcp232 [8/256/0/180] [ 766.880038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 766.897509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 766.911392] LNet: Removed LNI 192.168.203.12@tcp232 [ 766.919401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 766.929249] LNet: Added LNI 192.168.203.12@tcp232 [8/256/0/180] [ 767.204591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 767.218854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 767.235579] LNet: Removed LNI 192.168.203.12@tcp232 [ 767.242247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 769.346603] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 01:33:21 (1784093601) [ 769.697309] Key type .llcrypt unregistered [ 769.698738] Key type ._llcrypt unregistered [ 771.692728] Key type ._llcrypt registered [ 771.694449] Key type .llcrypt registered [ 771.758502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 771.763733] alg: No test for adler32 (adler32-zlib) [ 772.580476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 772.592251] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 775.567571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 778.459216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.473181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 778.483752] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 778.999724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_interrupt -r 2 [ 779.014729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_interrupt -r 2 [ 779.034110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_interrupt -r 2 [ 779.050825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_interrupt -r 2 [ 779.069379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_interrupt -r 2 [ 779.087106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_interrupt -r 2 [ 779.104567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_interrupt -r 2 [ 779.120398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_interrupt -r 2 [ 779.138212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp 192.168.203.112@tcp [ 779.148719] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 779.152388] LNet: 1 local NIs in recovery (showing 1): 192.168.203.12@tcp [ 779.236165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 779.249553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 779.262611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 779.276706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 779.290824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 779.306478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 779.323275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 779.338330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 779.347439] LNet: 87886:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 779.356582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 779.368637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 779.383297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 779.397864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 779.412921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 779.428642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 779.637515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_dropped -r 2 [ 779.655256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_dropped -r 2 [ 779.672757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_dropped -r 2 [ 779.690471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_dropped -r 2 [ 779.709475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_dropped -r 2 [ 779.729155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_dropped -r 2 [ 779.747366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_dropped -r 2 [ 779.765037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_dropped -r 2 [ 779.782251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp 192.168.203.112@tcp [ 779.792401] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 779.885295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 779.899427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 779.914726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 779.929267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 779.944227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 779.958397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 779.973729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 779.986948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 779.995990] LNet: 88006:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 780.005604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 780.020777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 780.035776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 780.050888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 780.066776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 780.082988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 780.299828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_aborted -r 2 [ 780.318562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_aborted -r 2 [ 780.337047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_aborted -r 2 [ 780.356216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_aborted -r 2 [ 780.374521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_aborted -r 2 [ 780.391661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_aborted -r 2 [ 780.409180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_aborted -r 2 [ 780.426382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_aborted -r 2 [ 780.442691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp 192.168.203.112@tcp [ 780.542880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 780.555282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 780.569184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 780.582578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 780.597821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 780.612070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 780.626883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 780.642324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 780.654647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 780.669478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 780.684493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 780.698365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 780.711672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 780.727601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 780.924750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_no_route -r 2 [ 780.942423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_no_route -r 2 [ 780.958893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_no_route -r 2 [ 780.975342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_no_route -r 2 [ 780.991320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_no_route -r 2 [ 781.009172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_no_route -r 2 [ 781.026938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_no_route -r 2 [ 781.043234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_no_route -r 2 [ 781.059910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp 192.168.203.112@tcp [ 781.070654] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 781.074535] LNet: Skipped 2 previous similar messages [ 781.163021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 781.176095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 781.189683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 781.204029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 781.218957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 781.233801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 781.248594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 781.264906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 781.273838] LNet: 88246:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 781.277237] LNet: 88246:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 781.289235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 781.305573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 781.323426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 781.339291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 781.355227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 781.371712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 781.578232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_timeout -r 2 [ 781.596441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_timeout -r 2 [ 781.613953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_timeout -r 2 [ 781.633941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_timeout -r 2 [ 781.654709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_timeout -r 2 [ 781.673934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_timeout -r 2 [ 781.692141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_timeout -r 2 [ 781.709554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_timeout -r 2 [ 781.726845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp 192.168.203.112@tcp [ 781.822166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 781.837852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 781.852601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 781.865636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 781.881425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 781.895089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 781.910106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 781.926368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 781.941688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 781.958525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 781.975765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 781.991903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 782.010181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 782.025806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 782.217931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e remote_dropped -r 2 [ 782.234429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e remote_dropped -r 2 [ 782.251364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e remote_dropped -r 2 [ 782.265310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e remote_dropped -r 2 [ 782.281096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e remote_dropped -r 2 [ 782.298693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e remote_dropped -r 2 [ 782.316824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e remote_dropped -r 2 [ 782.336148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e remote_dropped -r 2 [ 782.357745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp 192.168.203.112@tcp [ 782.370699] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.112@tcp [ 782.451443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 782.464021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 782.478466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 782.492626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 782.508247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 782.523781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 782.538787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 782.552932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 782.568576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 782.584328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 782.598972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 782.614100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 782.629813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 782.645367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 784.929515] LNet: 88560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.932104] LNetError: 88560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.934996] LNet: Removed LNI 192.168.203.12@tcp [ 784.944312] LNet: Removed LNI 192.168.203.12@tcp1 [ 785.091253] Key type .llcrypt unregistered [ 785.092872] Key type ._llcrypt unregistered [ 791.101969] Key type ._llcrypt registered [ 791.104709] Key type .llcrypt registered [ 791.621615] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 01:33:44 (1784093624) [ 791.769488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 791.775952] alg: No test for adler32 (adler32-zlib) [ 792.593843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.618809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 792.631688] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 792.973956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 792.988821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.12@tcp [ 792.999501] LNetError: 89537:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.203.12@tcp to recovery [ 793.033382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.12@tcp [ 793.080618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 795.331537] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 01:33:47 (1784093627) [ 795.375694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.385869] LNet: 90122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.389985] LNetError: 90122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.400799] LNet: Removed LNI 192.168.203.12@tcp [ 795.413852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.439245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 795.451835] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 795.749497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 795.759479] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 796.123776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.12@tcp [ 796.134415] LNetError: 90391:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.203.12@tcp to recovery [ 796.447319] LNet: 1 local NIs in recovery (showing 1): 192.168.203.12@tcp [ 796.447833] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 797.220537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.12@tcp [ 801.491824] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 01:33:54 (1784093634) [ 802.141530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg312-server [ 802.727810] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 01:33:55 (1784093635) [ 803.424930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg312-server [ 804.034104] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 01:33:56 (1784093636) [ 804.659783] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg312-server [ 805.245954] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 01:33:57 (1784093637) [ 805.278995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.289910] LNet: 92014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.293694] LNetError: 92014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.296945] LNet: Removed LNI 192.168.203.12@tcp [ 805.302736] LNet: Removed LNI 192.168.203.12@tcp1 [ 805.314270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.340531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 805.352484] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 805.639406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 805.670592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 805.693885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 805.727704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 805.747929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 805.767142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 808.097215] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 01:34:00 (1784093640) [ 808.139720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.150563] LNet: 92739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.154202] LNetError: 92739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.157726] LNetError: 92739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 808.174041] LNet: Removed LNI 192.168.203.12@tcp [ 808.187168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.215747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 808.230898] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 811.049712] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 01:34:03 (1784093643) [ 811.109263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.122689] LNet: 93447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.128572] LNetError: 93447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.138772] LNet: Removed LNI 192.168.203.12@tcp [ 811.151405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.178609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.190828] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 813.913680] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 01:34:06 (1784093646) [ 814.462729] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 815.069659] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 01:34:07 (1784093647) [ 815.240289] LNet: 94482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.243903] LNetError: 94482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.257746] LNet: Removed LNI 192.168.203.12@tcp [ 815.498601] Key type .llcrypt unregistered [ 815.500195] Key type ._llcrypt unregistered [ 817.708890] Key type ._llcrypt registered [ 817.710879] Key type .llcrypt registered [ 817.785967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 817.792953] alg: No test for adler32 (adler32-zlib) [ 818.608502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 818.623541] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 821.689969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 823.292721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.112@tcp [ 823.422449] LNet: 94897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.426897] LNetError: 94897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.436567] LNet: Removed LNI 192.168.203.12@tcp [ 823.603555] Key type .llcrypt unregistered [ 823.605357] Key type ._llcrypt unregistered [ 827.741753] Key type ._llcrypt registered [ 827.743863] Key type .llcrypt registered [ 828.262971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 828.839122] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 829.435859] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 01:34:21 (1784093661) [ 829.954723] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 830.498904] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 01:34:23 (1784093663) [ 831.149964] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg312-server [ 831.728166] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 01:34:24 (1784093664) [ 832.327506] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg312-server [ 832.901854] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 01:34:25 (1784093665) [ 833.367030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 833.425210] alg: No test for adler32 (adler32-zlib) [ 834.284485] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 834.371891] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 835.959163] Key type lgssc registered [ 836.414463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 844.582432] Key type lgssc unregistered [ 844.699323] LNet: 98070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.702570] LNetError: 98070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.709524] LNet: Removed LNI 192.168.203.12@tcp [ 844.964585] Key type .llcrypt unregistered [ 844.965927] Key type ._llcrypt unregistered [ 850.003556] Key type ._llcrypt registered [ 850.005199] Key type .llcrypt registered [ 850.484114] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 01:34:43 (1784093683) [ 850.606739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 850.612952] alg: No test for adler32 (adler32-zlib) [ 851.420754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.445280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 851.463434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 851.483460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 851.502480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 853.557896] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 01:34:46 (1784093686) [ 853.587922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 853.599715] LNet: 99476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.613631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.644033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 853.654570] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 853.976879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 853.984369] LNetError: 99625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.992282] LNet: Removed LNI 192.168.203.12@tcp [ 853.998915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.016520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 854.025981] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 854.193310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 854.201922] LNet: 99653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.205052] LNet: 99653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 854.214354] LNet: Removed LNI 192.168.203.12@tcp [ 854.222565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.246454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 854.260048] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 856.622357] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 01:34:49 (1784093689) [ 856.660521] LNet: 100237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.664230] LNetError: 100237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.668206] LNetError: 100237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 856.676852] LNet: Removed LNI 192.168.203.12@tcp [ 856.924844] Key type .llcrypt unregistered [ 856.926467] Key type ._llcrypt unregistered [ 859.166351] Key type ._llcrypt registered [ 859.168028] Key type .llcrypt registered [ 859.241817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 859.248469] alg: No test for adler32 (adler32-zlib) [ 860.066178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.104824] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 860.107129] LNet: *** cfs_fail_loc=e003, val=100*** [ 860.399648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 860.411620] Ping reply received with mapping: [ 860.413673] found metadata reply NI[192.168.203.12@tcp] [ 860.415187] 0c cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 860.418653] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 860.420913] 5a 5a 5a 5a 5a 5a 5a 5a [ 860.475356] LNet: 100641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.479477] LNetError: 100641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.487476] LNet: Removed LNI 192.168.203.12@tcp [ 860.734453] Key type .llcrypt unregistered [ 860.736517] Key type ._llcrypt unregistered [ 864.573351] Key type ._llcrypt registered [ 864.574783] Key type .llcrypt registered [ 865.038950] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 01:34:57 (1784093697) [ 865.217578] Key type .llcrypt unregistered [ 865.218945] Key type ._llcrypt unregistered [ 867.397838] Key type ._llcrypt registered [ 867.399701] Key type .llcrypt registered [ 867.470432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 867.477127] alg: No test for adler32 (adler32-zlib) [ 868.350372] LNetError: 101684:0:(api-ni.c:2795:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 868.491457] Key type .llcrypt unregistered [ 868.492739] Key type ._llcrypt unregistered [ 870.315566] Key type ._llcrypt registered [ 870.316696] Key type .llcrypt registered [ 870.823580] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 01:35:03 (1784093703) [ 871.035293] Key type .llcrypt unregistered [ 871.036945] Key type ._llcrypt unregistered [ 873.172252] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 873.174591] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 875.358939] Key type ._llcrypt registered [ 875.360411] Key type .llcrypt registered [ 875.435159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 875.442514] alg: No test for adler32 (adler32-zlib) [ 876.255424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.283088] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 880.371650] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 01:35:12 (1784093712) [ 880.406908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.418771] LNet: 103456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.422809] LNetError: 103456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 880.433731] LNet: Removed LNI 10.1.2.3@tcp [ 880.449418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.718489] Key type .llcrypt unregistered [ 880.720443] Key type ._llcrypt unregistered [ 882.767716] Key type ._llcrypt registered [ 882.769591] Key type .llcrypt registered [ 882.837455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 882.843580] alg: No test for adler32 (adler32-zlib) [ 883.647103] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 883.669476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 883.722952] LNet: 103749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.726741] LNetError: 103749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.738291] LNet: Removed LNI 192.168.203.12@tcp [ 883.957510] Key type .llcrypt unregistered [ 883.959044] Key type ._llcrypt unregistered [ 887.782981] Key type ._llcrypt registered [ 887.784175] Key type .llcrypt registered [ 888.261151] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 01:35:20 (1784093720) [ 888.477448] Key type .llcrypt unregistered [ 888.478634] Key type ._llcrypt unregistered [ 890.454716] Key type ._llcrypt registered [ 890.456235] Key type .llcrypt registered [ 890.529151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 890.535438] alg: No test for adler32 (adler32-zlib) [ 893.529544] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 01:35:26 (1784093726) [ 893.562558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.579253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.608498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 893.617979] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 893.864779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 893.881558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 893.902514] LNet: Removed LNI 192.168.203.12@tcp [ 893.992827] LNet: 105517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.133383] Key type .llcrypt unregistered [ 894.134442] Key type ._llcrypt unregistered [ 897.863164] Key type ._llcrypt registered [ 897.864778] Key type .llcrypt registered [ 898.336869] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 01:35:30 (1784093730) [ 898.460355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 898.466812] alg: No test for adler32 (adler32-zlib) [ 899.277491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.299916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 899.309087] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 899.572362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 899.586864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.12@tcp [ 899.595690] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 899.598442] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 899.600828] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 899, 0, 0 [ 899.603826] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 899.605935] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 899.608157] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 899.609770] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 899, 0, 0 [ 899.612075] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 899.613464] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 899.615534] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 899.617903] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 899, 0, 949 [ 899.621200] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 899.623498] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 899.626045] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 899.628000] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 899, 0, 949 [ 899.631155] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 901.699690] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 01:35:34 (1784093734) [ 901.849534] LNet: 107056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.852510] LNetError: 107056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.860830] LNet: Removed LNI 192.168.203.12@tcp [ 902.079356] Key type .llcrypt unregistered [ 902.080880] Key type ._llcrypt unregistered [ 903.927576] Key type ._llcrypt registered [ 903.929063] Key type .llcrypt registered [ 903.995153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 904.000903] alg: No test for adler32 (adler32-zlib) [ 904.804978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 904.818515] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 907.686977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 910.313548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.328699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 910.339060] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 910.655248] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 910.715797] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 910.720207] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 910.722174] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 912.832799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 912.841542] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 916.428585] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 01:35:49 (1784093749) [ 916.472749] LNet: 108553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.476701] LNetError: 108553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.480381] LNet: Removed LNI 192.168.203.12@tcp [ 917.535227] LNet: Removed LNI 192.168.203.12@tcp1 [ 917.536680] LNetError: 108553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 917.539382] LNetError: 108553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 917.547944] LNet: Removed LNI 10.1.2.3@tcp99 [ 917.774889] Key type .llcrypt unregistered [ 917.776427] Key type ._llcrypt unregistered [ 919.856105] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 919.858028] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 922.026979] Key type ._llcrypt registered [ 922.028559] Key type .llcrypt registered [ 922.087968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 922.092919] alg: No test for adler32 (adler32-zlib) [ 922.893777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.914658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 922.924098] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 923.178784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 923.187356] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 923.923862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 923.944416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 923.961061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 923.968943] LNetError: 109303:0:(peer.c:2276:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 923.982458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 923.996041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 924.127970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 924.154904] LNet: 109323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.157222] LNetError: 109323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 924.159809] LNet: Removed LNI 10.1.2.3@tcp [ 925.221283] LNet: Removed LNI 192.168.203.12@tcp [ 925.440513] Key type .llcrypt unregistered [ 925.441659] Key type ._llcrypt unregistered [ 928.917047] Key type ._llcrypt registered [ 928.918758] Key type .llcrypt registered [ 929.358830] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 01:36:01 (1784093761) [ 929.475217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 929.481295] alg: No test for adler32 (adler32-zlib) [ 930.282166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.303166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 930.312263] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 930.563919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 930.593759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg312-client@tcp [ 932.585957] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 01:36:05 (1784093765) [ 932.619888] LNet: 110848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.622908] LNetError: 110848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.631542] LNet: Removed LNI 192.168.203.12@tcp [ 932.882636] Key type .llcrypt unregistered [ 932.883970] Key type ._llcrypt unregistered [ 934.912108] Key type ._llcrypt registered [ 934.913159] Key type .llcrypt registered [ 934.965719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 934.972820] alg: No test for adler32 (adler32-zlib) [ 935.768190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 935.778874] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 935.789884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 937.959661] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 01:36:10 (1784093770) [ 938.509768] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 939.123513] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 01:36:11 (1784093771) [ 939.653957] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 940.290607] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 01:36:12 (1784093772) [ 940.337463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 940.349672] LNet: 112266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.354252] LNetError: 112266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.364963] LNet: Removed LNI 192.168.203.12@tcp [ 940.375240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.399239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 940.410986] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 940.694610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 940.718700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 943.072391] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 01:36:15 (1784093775) [ 943.110491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.123660] LNet: 112976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.127041] LNetError: 112976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.136522] LNet: Removed LNI 192.168.203.12@tcp [ 943.146635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.173476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 943.189376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 943.437560] Key type .llcrypt unregistered [ 943.439475] Key type ._llcrypt unregistered [ 947.380051] Key type ._llcrypt registered [ 947.381259] Key type .llcrypt registered [ 947.815938] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 01:36:20 (1784093780) [ 947.949661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 947.956258] alg: No test for adler32 (adler32-zlib) [ 948.754916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.774960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 948.784882] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 949.035315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 949.056209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 951.053705] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 01:36:23 (1784093783) [ 951.086161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.094909] LNet: 114528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.097574] LNetError: 114528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.110757] LNet: Removed LNI 192.168.203.12@tcp [ 951.121309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.143039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 951.155410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 953.110306] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 01:36:25 (1784093785) [ 953.142452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.151075] LNet: 115108:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.161255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.185412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 953.305343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 953.317183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 953.336379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 953.449221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 953.459184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 953.482920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.496356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.522538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 953.649389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 953.661282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 953.680702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 953.803575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 953.816655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 955.893902] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 01:36:28 (1784093788) [ 955.923863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 955.934235] LNet: 115738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.937438] LNet: 115738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 955.949930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.978431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 955.991966] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 956.265211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 956.275984] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 956.583914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 956.607391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 956.690502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.699872] LNetError: 116023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.702415] LNet: Removed LNI 192.168.203.12@tcp [ 956.707435] LNet: Removed LNI 192.168.203.12@tcp2 [ 956.717503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.740459] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 956.743807] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 956.746394] LNet: 116029:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 956.917390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 956.989243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.047175] LNet: Removed LNI 192.168.203.12@tcp [ 958.049058] LNetError: 116066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.051732] LNetError: 116066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 958.060400] LNet: Removed LNI 192.168.203.12@tcp2 [ 958.069074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.091535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 958.101614] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 958.376101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 958.384197] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 958.723284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 958.742440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 958.764229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 958.843939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.853588] LNet: 116351:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.857406] LNet: 116351:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 959.903178] LNet: Removed LNI 192.168.203.12@tcp [ 960.933395] LNet: Removed LNI 192.168.203.12@tcp2 [ 960.941387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.963923] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 960.966820] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 960.969401] LNet: 116357:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 961.145996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 961.225150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.235472] LNetError: 116394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.238727] LNetError: 116394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 961.241984] LNet: Removed LNI 192.168.203.12@tcp [ 962.277443] LNet: Removed LNI 192.168.203.12@tcp2 [ 962.286036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.307363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 962.316529] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 962.566340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 962.574815] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 962.863345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 962.884181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 962.954275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.963170] LNet: 116674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.965580] LNet: 116674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 963.999151] LNet: Removed LNI 192.168.203.12@tcp [ 964.006650] LNet: Removed LNI 192.168.203.12@tcp2 [ 964.013725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.037067] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 964.040293] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 964.042747] LNet: 116680:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 964.213937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 964.280993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.343098] LNet: Removed LNI 192.168.203.12@tcp [ 965.344843] LNetError: 116717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.347466] LNetError: 116717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 965.356297] LNet: Removed LNI 192.168.203.12@tcp2 [ 965.364566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.385610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 965.395216] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 965.630028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 965.637341] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 965.915477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 965.930069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 965.951418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 966.020510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.071123] LNet: Removed LNI 192.168.203.12@tcp [ 968.101297] LNet: Removed LNI 192.168.203.12@tcp2 [ 968.108700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.129979] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 968.132369] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 968.134379] LNet: 117007:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 968.292335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 968.360369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.368088] LNet: Removed LNI 192.168.203.12@tcp [ 969.381381] LNet: Removed LNI 192.168.203.12@tcp2 [ 969.389817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.411336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 969.422569] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 969.681088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 969.690544] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 970.003425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 970.018674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 970.042313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 970.170812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.232064] LNet: Removed LNI 192.168.203.12@tcp [ 971.239318] LNet: Removed LNI 192.168.203.12@tcp2 [ 971.246875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.269691] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 971.273090] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 971.436365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 971.543858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.551530] LNet: 117399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.554526] LNet: 117399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 972.575177] LNet: Removed LNI 192.168.203.12@tcp [ 972.582314] LNet: Removed LNI 192.168.203.12@tcp2 [ 972.590352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 972.610940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 972.620140] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 972.878855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 972.886393] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 973.179254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 973.194389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 973.204793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 973.226165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 973.334165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.367110] LNet: Removed LNI 192.168.203.12@tcp [ 974.368475] LNetError: 117700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.371625] LNetError: 117700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 974.381252] LNet: Removed LNI 192.168.203.12@tcp2 [ 974.388629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.411527] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 974.414556] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 974.416933] LNet: 117706:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 974.419790] LNet: 117706:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 974.585261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 976.772425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 977.293541] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 01:36:49 (1784093809) [ 977.321818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 977.330643] LNet: Removed LNI 192.168.203.12@tcp [ 978.341617] LNet: Removed LNI 192.168.203.12@tcp2 [ 978.349938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.462756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 978.471175] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 978.725521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 978.733740] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 979.060425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 979.068285] LNet: Removed LNI 10.1.2.3@tcp [ 979.075370] LNet: Removed LNI 192.168.203.12@tcp [ 979.082923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.105218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 979.115738] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 979.117307] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 979.251098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 979.262467] LNet: Removed LNI 192.168.203.12@tcp [ 979.274289] LNet: Removed LNI 10.1.2.3@tcp [ 981.289709] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 01:36:53 (1784093813) [ 981.318859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.338022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.375332] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 981.433723] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 981.443134] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 981.445488] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 983.555172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 983.565466] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 983.824575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 983.835097] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 984.594554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 984.611097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 984.635956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.12@tcp -m PUT -e local_timeout -r 1 [ 984.661539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 984.684520] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 984.687164] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 984.763980] LNetError: 119402:0:(lib-move.c:4771:lnet_parse()) 192.168.203.12@tcp, src 192.168.203.12@tcp: Dropping PUT (error -108 looking up sender) [ 985.823507] LNet: Removed LNI 10.1.2.3@tcp [ 986.853350] LNet: Removed LNI 192.168.203.12@tcp [ 986.999578] Key type .llcrypt unregistered [ 987.000997] Key type ._llcrypt unregistered [ 990.843807] Key type ._llcrypt registered [ 990.844855] Key type .llcrypt registered [ 991.297528] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 01:37:03 (1784093823) [ 991.423650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 991.429984] alg: No test for adler32 (adler32-zlib) [ 992.235398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.361836] LNet: 120713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.575307] Key type .llcrypt unregistered [ 992.576415] Key type ._llcrypt unregistered [ 994.483112] Key type ._llcrypt registered [ 994.484596] Key type .llcrypt registered [ 994.546471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 994.552477] alg: No test for adler32 (adler32-zlib) [ 995.351447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 995.362567] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 998.086818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 998.139125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 998.317203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.112@tcp [ 998.393541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 998.403520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 998.413484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 998.423193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 998.432070] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.112@tcp [ 998.433383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 998.444750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 998.457783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 998.471246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 998.478269] LNet: 121161:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 998.485691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 998.497161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 998.508741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 998.519191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 998.529942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 998.542093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 998.737074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 998.817063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 998.828179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 998.840923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 998.852951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 998.864983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 998.876839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 998.888917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 998.899595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 998.910561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 998.921379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 998.932653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 998.944193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 998.956211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 998.969042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 999.160919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.112@tcp [ 999.231499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 999.242455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 999.253552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 999.265565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 999.277400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 999.289211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 999.301858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 999.314415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 999.321531] LNet: 121395:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 999.323863] LNet: 121395:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 999.330704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 999.341213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 999.351642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 999.362525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 999.373417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 999.384495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 999.590465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 999.672876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 999.684621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 999.696791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 999.708174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 999.719601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 999.731046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 999.743985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 999.757594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 999.769551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 999.783649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 999.796950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 999.810459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 999.822711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 999.835045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 999.934697] LNet: 121558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.937411] LNetError: 121558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.944238] LNet: Removed LNI 192.168.203.12@tcp [ 1000.062775] Key type .llcrypt unregistered [ 1000.064096] Key type ._llcrypt unregistered [ 1005.039632] Key type ._llcrypt registered [ 1005.040640] Key type .llcrypt registered [ 1005.445055] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 01:37:18 (1784093838) [ 1005.558645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1005.563570] alg: No test for adler32 (adler32-zlib) [ 1006.356775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.472384] LNet: 122443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.670387] Key type .llcrypt unregistered [ 1006.671541] Key type ._llcrypt unregistered [ 1008.515032] Key type ._llcrypt registered [ 1008.516290] Key type .llcrypt registered [ 1008.572794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1008.578837] alg: No test for adler32 (adler32-zlib) [ 1009.379287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1009.391077] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 1012.047461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 1014.530760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.543971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1014.552272] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 1014.826067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1015.001168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.112@tcp [ 1015.078379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1015.091754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1015.103195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1015.114280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1015.126680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1015.136470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1015.146400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1015.156847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1015.163114] LNet: 123092:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1015.169851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1015.181109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1015.194090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1015.207520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1015.220555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1015.233889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1015.427470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1015.497646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1015.509228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1015.519079] LNet: 2 peer NIs in recovery (showing 2): 192.168.203.112@tcp, 192.168.203.112@tcp1 [ 1015.522729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1015.534267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1015.544014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1015.555321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1015.567463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1015.579839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1015.591163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1015.602553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1015.614873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1015.626874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1015.638774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1015.650558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1015.881596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.112@tcp [ 1015.980688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1015.995074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1016.008583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1016.021988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1016.033988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1016.046758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1016.059630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1016.070929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1016.079940] LNet: 123326:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1016.083074] LNet: 123326:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1016.090894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1016.103240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1016.115116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1016.127627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1016.139394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1016.152267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1016.356198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1016.452582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1016.465988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1016.479295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1016.494477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1016.508738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1016.521160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1016.533526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1016.546584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1016.560313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1016.572902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1016.585549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1016.599302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1016.612883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1016.624159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1018.651347] LNet: 123524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.653796] LNetError: 123524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.656959] LNet: Removed LNI 192.168.203.12@tcp [ 1018.669290] LNet: Removed LNI 192.168.203.12@tcp1 [ 1018.798600] Key type .llcrypt unregistered [ 1018.799542] Key type ._llcrypt unregistered [ 1023.568709] Key type ._llcrypt registered [ 1023.570014] Key type .llcrypt registered [ 1023.957757] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 01:37:36 (1784093856) [ 1024.412641] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 1024.944810] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 01:37:37 (1784093857) [ 1027.103761] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 01:37:39 (1784093859) [ 1029.204643] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 01:37:41 (1784093861) [ 1031.419432] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 01:37:44 (1784093864) [ 1033.483018] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 01:37:46 (1784093866) [ 1035.517656] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 01:37:48 (1784093868) [ 1037.778623] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 01:37:50 (1784093870) [ 1040.013443] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 01:37:52 (1784093872) [ 1042.174695] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 01:37:54 (1784093874) [ 1044.279420] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 01:37:56 (1784093876) [ 1046.442156] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 01:37:59 (1784093879) [ 1048.648267] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 01:38:01 (1784093881) [ 1050.721185] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 01:38:03 (1784093883) [ 1052.863086] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 01:38:05 (1784093885) [ 1055.043955] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 01:38:07 (1784093887) [ 1057.185688] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 01:38:09 (1784093889) [ 1059.269474] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 01:38:11 (1784093891) [ 1061.501568] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 01:38:14 (1784093894) [ 1061.642602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1061.647937] alg: No test for adler32 (adler32-zlib) [ 1062.455881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.501161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1062.509942] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 1062.876377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1062.883712] LNet: 134503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.887261] LNetError: 134503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.896310] LNet: Removed LNI 192.168.203.12@tcp [ 1062.903867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.926766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1062.937092] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 1065.187183] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 01:38:17 (1784093897) [ 1067.378531] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 01:38:20 (1784093900) [ 1069.561579] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 01:38:22 (1784093902) [ 1071.916279] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 01:38:24 (1784093904) [ 1074.175891] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 01:38:26 (1784093906) [ 1076.465000] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 01:38:29 (1784093909) [ 1078.860975] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 01:38:31 (1784093911) [ 1081.104704] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 01:38:33 (1784093913) [ 1083.372767] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 01:38:35 (1784093915) [ 1085.701731] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 01:38:38 (1784093918) [ 1087.723892] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 01:38:40 (1784093920) [ 1089.689490] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 01:38:42 (1784093922) [ 1091.835555] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 947 sec =========== 01:38:44 (1784093924) [ 1091.925712] LNet: 141957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.929335] LNetError: 141957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1091.939495] LNet: Removed LNI 192.168.203.12@tcp [ 1092.181494] Key type .llcrypt unregistered [ 1092.182428] Key type ._llcrypt unregistered [ 1096.126290] Key type ._llcrypt registered [ 1096.127967] Key type .llcrypt registered [ 1096.298958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1096.304170] alg: No test for adler32 (adler32-zlib) [ 1097.157629] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 1097.240494] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 1098.831143] Key type lgssc registered [ 1099.200701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1120.290659] Lustre: Mounted lustre-client [ 1122.752650] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1125.003112] Lustre: Unmounted lustre-client [ 1136.481410] Key type lgssc unregistered [ 1136.613431] LNet: 144871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.616310] LNetError: 144871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.624440] LNet: Removed LNI 192.168.203.12@tcp [ 1136.890164] Key type .llcrypt unregistered [ 1136.891807] Key type ._llcrypt unregistered