[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 423655804 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002327] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.005008] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010119] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013082] *** VALIDATE selinux *** [ 0.022236] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027110] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030015] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031148] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032147] *** VALIDATE tmpfs *** [ 0.034394] *** VALIDATE proc *** [ 0.035304] *** VALIDATE cgroup *** [ 0.037013] *** VALIDATE cgroup2 *** [ 0.038361] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039182] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041042] Spectre V2 : User space: Vulnerable [ 0.042011] Speculative Store Bypass: Vulnerable [ 0.045488] debug: unmapping init [mem 0xffffffffb6a59000-0xffffffffb6a60fff] [ 0.047216] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048800] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049029] ... version: 2 [ 0.050014] ... bit width: 48 [ 0.051014] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053020] ... max period: 00007fffffffffff [ 0.054019] ... fixed-purpose events: 3 [ 0.055012] ... event mask: 000000070000000f [ 0.057281] rcu: Hierarchical SRCU implementation. [ 0.059800] smp: Bringing up secondary CPUs ... [ 0.060721] x86: Booting SMP configuration: [ 0.061032] .... node #0, CPUs: #1 #2 #3 [ 0.072108] smp: Brought up 1 node, 4 CPUs [ 0.074022] smpboot: Max logical packages: 1 [ 0.075017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.240041] node 0 deferred pages initialised in 162ms [ 0.244164] devtmpfs: initialized [ 0.245295] x86/mm: Memory block size: 128MB [ 0.249147] gcov: version magic: 0x41383552 [ 0.251258] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.254068] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.256358] pinctrl core: initialized pinctrl subsystem [ 0.258153] [ 0.258602] ************************************************************* [ 0.260009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262010] ** ** [ 0.263008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.265014] ** ** [ 0.268014] ** This means that this kernel is built to expose internal ** [ 0.270013] ** IOMMU data structures, which may compromise security on ** [ 0.272014] ** your system. ** [ 0.274015] ** ** [ 0.276015] ** If you see this message and you are not debugging the ** [ 0.278013] ** kernel, report this immediately to your vendor! ** [ 0.280014] ** ** [ 0.282015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.283008] ************************************************************* [ 0.285754] NET: Registered protocol family 16 [ 0.287429] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.290081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.293061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.297109] cpuidle: using governor menu [ 0.298601] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.300464] PCI: Using configuration type 1 for base access [ 0.303130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.313111] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.316043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.319230] cryptd: max_cpu_qlen set to 1000 [ 0.321588] ACPI: Added _OSI(Module Device) [ 0.322000] ACPI: Added _OSI(Processor Device) [ 0.324015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.326014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.331078] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.336552] ACPI: Interpreter enabled [ 0.337059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.338009] ACPI: Using IOAPIC for interrupt routing [ 0.338914] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.341326] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.351139] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.353028] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.354010] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.356060] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.359787] acpiphp: Slot [2] registered [ 0.360000] acpiphp: Slot [5] registered [ 0.360000] acpiphp: Slot [6] registered [ 0.361085] acpiphp: Slot [3] registered [ 0.362087] acpiphp: Slot [4] registered [ 0.363149] acpiphp: Slot [7] registered [ 0.365090] acpiphp: Slot [8] registered [ 0.366101] acpiphp: Slot [9] registered [ 0.367124] acpiphp: Slot [10] registered [ 0.368150] acpiphp: Slot [11] registered [ 0.369184] acpiphp: Slot [12] registered [ 0.371112] acpiphp: Slot [13] registered [ 0.373205] acpiphp: Slot [14] registered [ 0.374098] acpiphp: Slot [15] registered [ 0.376093] acpiphp: Slot [16] registered [ 0.377153] acpiphp: Slot [17] registered [ 0.379093] acpiphp: Slot [18] registered [ 0.380101] acpiphp: Slot [19] registered [ 0.381128] acpiphp: Slot [20] registered [ 0.382000] acpiphp: Slot [21] registered [ 0.382000] acpiphp: Slot [22] registered [ 0.383192] acpiphp: Slot [23] registered [ 0.385099] acpiphp: Slot [24] registered [ 0.386106] acpiphp: Slot [25] registered [ 0.387137] acpiphp: Slot [26] registered [ 0.389137] acpiphp: Slot [27] registered [ 0.390108] acpiphp: Slot [28] registered [ 0.392103] acpiphp: Slot [29] registered [ 0.393097] acpiphp: Slot [30] registered [ 0.395171] acpiphp: Slot [31] registered [ 0.396087] PCI host bridge to bus 0000:00 [ 0.398028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.400026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.404036] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.408030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.410024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.413023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.415210] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.417735] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.420619] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.427014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.431054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.434019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.436015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.438020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.441665] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.445164] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.447031] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.449882] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.454013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.463855] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.470020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.474321] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.481015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.491019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.517016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.527000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.536016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.544015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.559016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.573000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.575620] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.578442] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.580701] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.583296] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.588105] iommu: Default domain type: Passthrough [ 0.589487] SCSI subsystem initialized [ 0.591152] ACPI: bus type USB registered [ 0.592211] usbcore: registered new interface driver usbfs [ 0.594067] usbcore: registered new interface driver hub [ 0.597096] usbcore: registered new device driver usb [ 0.599184] pps_core: LinuxPPS API ver. 1 registered [ 0.600012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.604063] PTP clock support registered [ 0.606136] EDAC MC: Ver: 3.0.0 [ 0.608101] PCI: Using ACPI for IRQ routing [ 0.609000] NetLabel: Initializing [ 0.611015] NetLabel: domain hash size = 128 [ 0.613015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.615084] NetLabel: unlabeled traffic allowed by default [ 0.617277] vgaarb: loaded [ 0.619084] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.620010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.630000] clocksource: Switched to clocksource kvm-clock [ 0.750135] VFS: Disk quotas dquot_6.6.0 [ 0.752254] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.758041] *** VALIDATE ramfs *** [ 0.759923] *** VALIDATE hugetlbfs *** [ 0.762449] pnp: PnP ACPI init [ 0.765287] pnp: PnP ACPI: found 6 devices [ 0.782587] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.786154] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.788359] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.791067] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.793503] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.796180] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.799285] NET: Registered protocol family 2 [ 0.801862] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.807184] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.811247] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.819199] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.822926] TCP: Hash tables configured (established 65536 bind 65536) [ 0.826306] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.829934] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.833342] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.837271] NET: Registered protocol family 1 [ 0.840581] RPC: Registered named UNIX socket transport module. [ 0.843759] RPC: Registered udp transport module. [ 0.845909] RPC: Registered tcp transport module. [ 0.848041] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.851274] NET: Registered protocol family 44 [ 0.853285] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.855819] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.858341] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.861170] PCI: CLS 0 bytes, default 64 [ 0.863387] Unpacking initramfs... [ 2.418488] debug: unmapping init [mem 0xffff8aea3cc64000-0xffff8aea3ffcffff] [ 2.423270] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.425741] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.428875] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.421381] Initialise system trusted keyrings [ 3.423340] Key type blacklist registered [ 3.425402] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.436767] zbud: loaded [ 3.440101] *** VALIDATE nfs *** [ 3.441505] *** VALIDATE nfs4 *** [ 3.443265] pstore: using deflate compression [ 3.447420] Platform Keyring initialized [ 3.569404] NET: Registered protocol family 38 [ 3.571046] Key type asymmetric registered [ 3.580276] Asymmetric key parser 'x509' registered [ 3.581772] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.584590] io scheduler mq-deadline registered [ 3.586253] io scheduler kyber registered [ 3.587696] io scheduler bfq registered [ 3.589425] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.592148] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.595162] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.597676] ACPI: Power Button [PWRF] [ 3.603604] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.610318] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.635856] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.679136] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.709145] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.714759] Non-volatile memory driver v1.3 [ 3.717444] Linux agpgart interface v0.103 [ 3.857251] virtio_blk virtio1: [vda] 134776 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.860276] vda: detected capacity change from 0 to 69005312 [ 4.076561] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.095326] vdb: detected capacity change from 0 to 1073741824 [ 4.492727] libphy: Fixed MDIO Bus: probed [ 4.532642] usbcore: registered new interface driver usbserial_generic [ 4.535240] usbserial: USB Serial support registered for generic [ 4.538951] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.542921] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.544694] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.557717] mousedev: PS/2 mouse device common for all mice [ 4.561752] rtc_cmos 00:05: RTC can wake from S4 [ 4.567843] rtc_cmos 00:05: registered as rtc0 [ 4.579015] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.597698] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.658608] intel_pstate: CPU model not supported [ 4.813159] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.825202] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.920559] hid: raw HID events driver (C) Jiri Kosina [ 4.927156] usbcore: registered new interface driver usbhid [ 4.947155] usbhid: USB HID core driver [ 4.986845] drop_monitor: Initializing network drop monitor service [ 4.990968] Initializing XFRM netlink socket [ 4.993172] NET: Registered protocol family 10 [ 4.997943] Segment Routing with IPv6 [ 4.999280] NET: Registered protocol family 17 [ 5.001687] mpls_gso: MPLS GSO support [ 5.006655] RAS: Correctable Errors collector initialized. [ 5.008693] AVX version of gcm_enc/dec engaged. [ 5.010387] AES CTR mode by8 optimization enabled [ 5.185259] sched_clock: Marking stable (5185018880, 0)->(6072672024, -887653144) [ 5.189288] registered taskstats version 1 [ 5.191710] Loading compiled-in X.509 certificates [ 5.194438] zswap: loaded using pool lzo/zbud [ 5.259618] Key type big_key registered [ 5.276771] Key type encrypted registered [ 5.278660] ima: No TPM chip found, activating TPM-bypass! [ 5.281219] ima: Allocated hash algorithm: sha1 [ 5.282958] ima: No architecture policies found [ 5.284849] evm: Initialising EVM extended attributes: [ 5.286818] evm: security.selinux [ 5.288268] evm: security.ima [ 5.289436] evm: security.capability [ 5.290853] evm: HMAC attrs: 0x1 [ 5.295645] rtc_cmos 00:05: setting system clock to 2026-05-25 07:08:35 UTC (1779692915) [ 5.307532] debug: unmapping init [mem 0xffffffffb7a03000-0xffffffffb7bfffff] [ 5.310990] debug: unmapping init [mem 0xffffffffb6782000-0xffffffffb6a58fff] [ 5.320309] Write protecting the kernel read-only data: 28672k [ 5.325445] debug: unmapping init [mem 0xffffffffb4e03000-0xffffffffb4ffffff] [ 5.329842] debug: unmapping init [mem 0xffffffffb5714000-0xffffffffb57fffff] [ 5.404646] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.412917] systemd[1]: Detected virtualization kvm. [ 5.415076] systemd[1]: Detected architecture x86-64. [ 5.417118] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.456996] systemd[1]: No hostname configured. [ 5.458777] systemd[1]: Set hostname to . [ 5.461293] random: systemd: uninitialized urandom read (16 bytes read) [ 5.463718] systemd[1]: Initializing machine ID from random generator. [ 5.576782] random: ln: uninitialized urandom read (6 bytes read) [ 5.757996] random: systemd: uninitialized urandom read (16 bytes read) [ 5.761454] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.774588] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.787403] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.343421] device-mapper: uevent: version 1.0.3 [ 7.351528] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] [ 9.853512] random: fast init done Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 10.146415] virtio_net virtio0 ens2: renamed from eth0 [ 10.718789] scsi host0: ata_piix [ 10.743190] scsi host1: ata_piix [ 10.745038] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 10.747568] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 15.822830] random: crng init done [ 15.830942] random: 7 urandom warning(s) missed due to ratelimiting [ 18.503031] dracut-initqueue[584]: RTNETLINK answers: File exists [ 19.385132] hrtimer: interrupt took 2521726 ns Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 21.402655] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.477417] printk: systemd: 25 output lines suppressed due to ratelimiting [ 28.063696] SELinux: Disabled at runtime. [ 28.325945] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 28.352899] systemd[1]: Detected virtualization kvm. [ 28.360785] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 31.326791] systemd[1]: initrd-switch-root.service: Succeeded. [ 31.341799] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 31.392616] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 31.407514] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 31.427566] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 31.465865] systemd[1]: Starting Journal Service... Starting Journal Service... [ 31.512153] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ 32.583329] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 35.078728] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 36.534529] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 36.704382] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 37.596649] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 37.703139] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (11s / no limit)[ 42.246426] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit)[ 43.523343] NFS: Registering the id_resolver key type [ 43.528533] Key type id_resolver registered [ 43.531794] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg408-client login: [ 123.672892] libcfs: loading out-of-tree module taints kernel. [ 123.980803] Key type ._llcrypt registered [ 123.996066] Key type .llcrypt registered [ 124.826945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 124.838610] alg: No test for adler32 (adler32-zlib) [ 127.071766] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 128.210849] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 130.051587] Key type lgssc registered [ 131.952988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 258.463379] Lustre: Mounted lustre-client [ 263.384388] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 275.840373] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing check_logdir /tmp/testlogs/ [ 281.380744] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing yml_node [ 284.135028] Lustre: lustre-OST0000-osc-ffff8aea88512000: disconnect after 24s idle [ 286.973988] Lustre: DEBUG MARKER: Client: 2.17.53.24 [ 289.410912] Lustre: DEBUG MARKER: MDS: 2.17.53.24 [ 292.294349] Lustre: DEBUG MARKER: OSS: 2.17.53.24 [ 293.742476] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon May 25 03:13:22 EDT 2026 [ 312.071319] LustreError: 7936:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8aea88512000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 312.148515] Lustre: Unmounted lustre-client [ 332.144186] Key type lgssc unregistered [ 332.415027] LNet: 8382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.422817] LNetError: 8382:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 332.456391] LNet: Removed LNI 192.168.204.8@tcp [ 333.245398] Key type .llcrypt unregistered [ 333.250227] Key type ._llcrypt unregistered [ 356.716321] Key type ._llcrypt registered [ 356.718757] Key type .llcrypt registered [ 356.948749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 356.970481] alg: No test for adler32 (adler32-zlib) [ 357.919553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 357.988501] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 358.013822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 362.314765] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 362.434499] LNet: 9272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.448746] LNetError: 9272:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 362.468888] LNet: Removed LNI 192.168.204.8@tcp [ 363.202548] Key type .llcrypt unregistered [ 363.204149] Key type ._llcrypt unregistered [ 369.810279] Key type ._llcrypt registered [ 369.817350] Key type .llcrypt registered [ 371.102705] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 03:14:39 (1779693279) [ 371.430985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 371.452146] alg: No test for adler32 (adler32-zlib) [ 372.395767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.506845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.533494] LNet: 9690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.565420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.637539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 372.673432] LNet: 9697:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 379.564741] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 03:14:48 (1779693288) [ 379.785944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.805835] LNet: 10274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.823932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.900746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 385.978477] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 03:14:54 (1779693294) [ 386.136865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.172818] LNet: 10860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.211869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.309615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 393.388981] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 03:15:02 (1779693302) [ 393.515514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.539665] LNet: 11440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.571462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.692420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 400.767458] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 03:15:09 (1779693309) [ 400.875673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.903880] LNet: 12024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.935763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.027989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 401.625893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 409.108978] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 03:15:17 (1779693317) [ 409.546292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.573722] LNet: 12620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.594790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.643968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 410.112355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 410.679295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 418.556973] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 03:15:26 (1779693326) [ 418.721396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.801440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.930288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 426.345979] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 03:15:34 (1779693334) [ 426.550548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.610206] LNet: 13802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.615607] LNet: 13802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 426.656508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.770417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 426.845960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 427.495095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 427.567989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 428.225785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 428.275621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 428.865823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 428.931053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 429.700831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 429.764073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 430.416742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 430.465431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 431.038803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 431.087665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 431.642265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 431.682276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 432.310205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 432.374626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 433.018925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 433.060167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 433.764382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 433.827858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 434.292407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 434.358111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 434.867547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 434.925441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 442.164136] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 03:15:50 (1779693350) [ 442.326894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.394239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.494530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 442.535042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 449.720582] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 03:15:58 (1779693358) [ 449.951537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.028566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.131060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 450.216725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 457.376161] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 03:16:06 (1779693366) [ 457.456794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.488176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.583732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 457.614307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 463.807382] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 03:16:12 (1779693372) [ 463.912339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.950426] LNet: 16235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 463.957686] LNet: 16235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 463.989218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.113112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 464.146639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 470.781496] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 03:16:19 (1779693379) [ 470.900932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.958467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.078506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 471.127439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 478.500954] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 03:16:27 (1779693387) [ 478.614522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.666211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.786706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 478.847922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 485.479576] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 03:16:34 (1779693394) [ 485.565612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.631684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.770938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.355043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 493.753131] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 03:16:42 (1779693402) [ 493.875965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.947386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.190445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 494.457631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 501.839958] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 03:16:50 (1779693410) [ 501.952202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.007758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.098650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 502.345692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 510.290639] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 03:16:58 (1779693418) [ 510.398091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.451638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.058502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 511.436932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 519.972247] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 03:17:08 (1779693428) [ 520.109067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.201514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.332691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 520.395754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 528.810733] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 03:17:17 (1779693437) [ 528.936270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.012216] LNet: 21070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 529.025947] LNet: 21070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 529.083351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.171761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 529.276540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 536.403442] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 03:17:25 (1779693445) [ 536.609511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.646340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.706570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 543.170676] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 03:17:31 (1779693451) [ 543.256447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.311798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.390935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 549.422388] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 03:17:38 (1779693458) [ 549.508170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.552853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.628115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 550.040347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 556.246376] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 03:17:44 (1779693464) [ 556.355870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.424624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.528444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 556.561228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 562.765278] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 03:17:51 (1779693471) [ 562.877181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.950690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.092606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 563.118600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 569.980647] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 03:17:58 (1779693478) [ 570.165788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.225852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.313727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 570.351700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 577.251312] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 03:18:05 (1779693485) [ 577.324528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.364134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.437738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 577.477994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 583.576966] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 03:18:12 (1779693492) [ 583.854644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.902273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.953851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 584.010766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 584.034068] LNetError: 25802:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 584.086914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 584.123758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 590.391918] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 03:18:19 (1779693499) [ 590.478764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.530362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.626761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8V9MZcLm3L [ 590.693680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8V9MZcLm3L [ 590.744612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8V9MZcLm3L [ 596.446393] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 03:18:25 (1779693505) [ 596.657325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.697951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.751568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 596.787518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 596.821579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 602.659499] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 03:18:31 (1779693511) [ 602.999429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.044873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.111747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 606.151126] LNet: Message forwarding will be enabled in 2s [ 609.247124] LNet: Message forwarding enabled [ 609.699976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 609.729429] LNet: Message forwarding will stop in 2s to 6s [ 614.235976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 614.642140] LNet: Message forwarding disabled [ 616.566851] LNet: Message forwarding will be enabled in 2s [ 619.744678] LNet: Message forwarding enabled [ 623.185437] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 03:18:52 (1779693532) [ 623.445462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.640542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.716729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 630.587191] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 03:18:59 (1779693539) [ 630.720841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.799093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.855678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 630.883983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 630.913626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 630.942868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 630.981667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 631.015220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 631.047317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 631.098847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 631.141645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 631.183946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 631.241397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 631.302898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 631.344785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 631.388863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 631.428469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 631.474907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 631.509805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 631.555421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 631.619922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 631.644078] LNetError: 28872:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 637.467440] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 03:19:06 (1779693546) [ 637.757230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.799689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.875377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 644.273989] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 03:19:13 (1779693553) [ 646.278812] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg408-server [ 648.132855] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 03:19:17 (1779693557) [ 648.383994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.429828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.466395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 648.493643] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 649.854778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[9-13]@tcp [ 649.878883] LNet: 30589:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 649.953324] LNetError: 30378:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.12@tcp is being used as a gateway but routing feature is not turned on [ 649.976942] LNetError: 30378:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.12@tcp has gone from up to down [ 650.918617] LNetError: 30378:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.9@tcp is being used as a gateway but routing feature is not turned on [ 650.928714] LNetError: 30378:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 651.956701] LNetError: 30378:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.9@tcp is being used as a gateway but routing feature is not turned on [ 651.969771] LNetError: 30378:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 653.992150] LNetError: 30378:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.13@tcp is being used as a gateway but routing feature is not turned on [ 654.011312] LNetError: 30378:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 656.078636] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 03:19:25 (1779693565) [ 657.928149] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg408-server [ 658.081549] LNetError: 30378:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.9@tcp is being used as a gateway but routing feature is not turned on [ 658.093394] LNetError: 30378:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 660.229393] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 03:19:28 (1779693568) [ 660.423791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.443671] LNet: 31523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.448541] LNet: 31523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 15 previous similar messages [ 660.453815] LNetError: 31523:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 660.473913] LNet: Removed LNI 192.168.204.8@tcp [ 660.505963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.572978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 660.602771] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 661.423415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[9-14/2]@tcp [ 661.441447] LNet: 31640:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 661.447835] LNet: 31640:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 661.470329] LNetError: 31527:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.9@tcp has gone from up to down [ 661.472709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[9-14/2]@tcp [ 661.486557] LNetError: 31527:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 668.863687] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 03:19:37 (1779693577) [ 669.046870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.092856] LNetError: 32230:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 669.117226] LNet: Removed LNI 192.168.204.8@tcp [ 669.162764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.293628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 669.386093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.445357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.565352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 669.650519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.697408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.748405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 669.844317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.900159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.991952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.117790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.155164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.268840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 670.437538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.517567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.628734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.775796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.836478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.954408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 671.069566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.118684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.221511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 671.299973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.341389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.405513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 671.512624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.596147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.692348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 671.829505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.930357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.070943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 672.201117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.242284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.342984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 678.444782] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 03:19:47 (1779693587) [ 680.411543] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg408-server [ 682.276311] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 03:19:50 (1779693590) [ 684.128775] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg408-server [ 686.530932] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 03:19:54 (1779693594) [ 687.669929] Key type .llcrypt unregistered [ 687.671589] Key type ._llcrypt unregistered [ 694.255523] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 694.413119] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 694.418931] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 694.425448] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 696.999626] Key type ._llcrypt registered [ 697.001867] Key type .llcrypt registered [ 697.224921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 697.241824] alg: No test for adler32 (adler32-zlib) [ 698.206803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.301226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 698.335112] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 698.953813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 698.989391] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 700.908669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 700.944047] LNet: Removed LNI 10.1.2.3@tcp [ 701.094878] LNet: 34349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.099886] LNetError: 34349:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.121781] LNet: Removed LNI 192.168.204.8@tcp [ 701.794677] Key type .llcrypt unregistered [ 701.797726] Key type ._llcrypt unregistered [ 713.990818] Key type ._llcrypt registered [ 713.995456] Key type .llcrypt registered [ 715.359595] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 03:20:24 (1779693624) [ 716.085778] Key type .llcrypt unregistered [ 716.089453] Key type ._llcrypt unregistered [ 722.622267] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 722.706772] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 722.714616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 722.717144] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 725.485803] Key type ._llcrypt registered [ 725.490231] Key type .llcrypt registered [ 725.675421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 725.691333] alg: No test for adler32 (adler32-zlib) [ 726.650961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.751504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 726.805191] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 727.556594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 727.581882] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 730.410219] LNet: 35844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.418457] LNetError: 35844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.428871] LNet: Removed LNI 192.168.204.8@tcp [ 730.450749] LNet: Removed LNI 10.1.2.3@tcp [ 731.188598] Key type .llcrypt unregistered [ 731.193093] Key type ._llcrypt unregistered [ 741.852766] Key type ._llcrypt registered [ 741.858615] Key type .llcrypt registered [ 743.093566] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 03:20:51 (1779693651) [ 743.767413] Key type .llcrypt unregistered [ 743.772367] Key type ._llcrypt unregistered [ 750.243209] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 750.383071] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 750.396829] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 750.406930] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 753.034588] Key type ._llcrypt registered [ 753.039865] Key type .llcrypt registered [ 753.280871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 753.302679] alg: No test for adler32 (adler32-zlib) [ 754.256577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.307650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 754.329927] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 756.124306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 756.158391] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 758.481652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 758.509868] LNet: Removed LNI 10.1.2.3@tcp [ 758.533614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 758.576088] LNet: Removed LNI 10.1.2.31@tcp [ 758.601340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 758.624540] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 759.495953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 759.519545] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 760.450555] LNet: 37766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.458174] LNetError: 37766:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 760.464836] LNet: Removed LNI 10.1.2.3@tcp [ 760.477374] LNet: Removed LNI 10.1.2.31@tcp [ 761.383648] Key type .llcrypt unregistered [ 761.389720] Key type ._llcrypt unregistered [ 775.797503] Key type ._llcrypt registered [ 775.802679] Key type .llcrypt registered [ 777.132185] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 03:21:25 (1779693685) [ 777.709982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 777.740720] alg: No test for adler32 (adler32-zlib) [ 778.647543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.695445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 778.713238] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 779.687883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.706728] LNet: 38721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.712921] LNetError: 38721:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.722816] LNet: Removed LNI 192.168.204.8@tcp [ 779.739819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.794281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.8@tcp [ 779.815929] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 780.508547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.8@tcp [ 780.540159] LNet: Removed LNI 192.168.204.8@tcp [ 780.604524] LNet: 38856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.319632] Key type .llcrypt unregistered [ 781.321462] Key type ._llcrypt unregistered [ 792.476452] Key type ._llcrypt registered [ 792.483207] Key type .llcrypt registered [ 793.679545] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 03:21:42 (1779693702) [ 795.440408] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg408-server [ 797.112946] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 03:21:46 (1779693706) [ 797.587546] Key type .llcrypt unregistered [ 797.589318] Key type ._llcrypt unregistered [ 803.176079] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 803.344704] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 803.361471] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 803.368379] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 805.958644] Key type ._llcrypt registered [ 805.961716] Key type .llcrypt registered [ 806.114821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 806.128592] alg: No test for adler32 (adler32-zlib) [ 807.010604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.059711] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 807.739437] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 810.103194] LNet: 40689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.120970] LNetError: 40689:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.133207] LNet: Removed LNI 192.168.204.8@tcp1 [ 810.153452] LNet: Removed LNI 10.1.2.3@tcp2 [ 810.901418] Key type .llcrypt unregistered [ 810.906936] Key type ._llcrypt unregistered [ 823.654697] Key type ._llcrypt registered [ 823.656884] Key type .llcrypt registered [ 825.463425] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 03:22:13 (1779693733) [ 825.927035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 825.975929] alg: No test for adler32 (adler32-zlib) [ 827.042116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.167431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 827.219388] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 828.162928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 828.197185] LNet: 41657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.204335] LNetError: 41657:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 828.225916] LNet: Removed LNI 192.168.204.8@tcp [ 828.253695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.387983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 828.457572] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 835.791359] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 03:22:24 (1779693744) [ 836.097978] LNet: 42248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.115602] LNetError: 42248:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 836.133068] LNet: Removed LNI 192.168.204.8@tcp [ 836.783532] Key type .llcrypt unregistered [ 836.790353] Key type ._llcrypt unregistered [ 843.928277] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 844.081153] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 844.085135] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 844.090972] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 846.666157] Key type ._llcrypt registered [ 846.669185] Key type .llcrypt registered [ 846.823410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 846.837406] alg: No test for adler32 (adler32-zlib) [ 847.802758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.863679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 847.886446] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 848.534031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 848.552068] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 850.809413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 850.825279] LNet: 42994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.832716] LNetError: 42994:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.837685] LNet: Removed LNI 192.168.204.8@tcp [ 850.850667] LNet: Removed LNI 10.1.2.3@tcp [ 850.908386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.989418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 851.031423] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 851.037813] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 851.482673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.509659] LNet: 43021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.519991] LNetError: 43021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.530405] LNetError: 43021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 851.539544] LNet: Removed LNI 192.168.204.8@tcp [ 851.554461] LNet: Removed LNI 10.1.2.3@tcp [ 851.583873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.654567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 851.684268] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 851.687618] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 852.280681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 852.308909] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 853.474909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 853.515623] LNet: 43183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.525854] LNetError: 43183:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.533627] LNetError: 43183:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 853.549197] LNet: Removed LNI 192.168.204.8@tcp [ 853.551841] LNet: Removed LNI 10.1.2.3@tcp [ 853.564625] LNet: Removed LNI 192.168.204.8@tcp2 [ 853.608145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.704784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 853.743139] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 853.755649] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 853.761036] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 862.200070] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 03:22:50 (1779693770) [ 862.343371] LNet: 43774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.350504] LNetError: 43774:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.358203] LNetError: 43774:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 862.369248] LNet: Removed LNI 192.168.204.8@tcp [ 862.371669] LNet: Removed LNI 10.1.2.3@tcp [ 862.380263] LNet: Removed LNI 192.168.204.8@tcp2 [ 863.393787] Key type .llcrypt unregistered [ 863.396407] Key type ._llcrypt unregistered [ 870.230343] Key type ._llcrypt registered [ 870.236069] Key type .llcrypt registered [ 870.569531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 870.598652] alg: No test for adler32 (adler32-zlib) [ 871.609169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.690937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 871.741192] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 872.868934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.911425] LNet: 44175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.931870] LNetError: 44175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.954437] LNet: Removed LNI 192.168.204.8@tcp [ 872.993519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.091286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 873.148134] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 873.875879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.902919] LNet: 44205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.917443] LNetError: 44205:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.940753] LNet: Removed LNI 192.168.204.8@tcp [ 873.956751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.028719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 874.080278] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 874.713315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.753539] LNet: Removed LNI 192.168.204.8@tcp [ 874.770312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.823594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 874.854758] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 875.228052] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 875.362864] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 875.366115] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 875.370166] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 877.857446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.871745] LNet: 44314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.874958] LNet: 44314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 877.882738] LNetError: 44314:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.887689] LNetError: 44314:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 877.897786] LNet: Removed LNI 192.168.204.8@tcp [ 877.908585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.969505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 877.994136] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 885.690486] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 03:23:14 (1779693794) [ 886.140033] LNet: 44911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.147083] LNetError: 44911:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.170453] LNet: Removed LNI 192.168.204.8@tcp [ 887.150437] Key type .llcrypt unregistered [ 887.152530] Key type ._llcrypt unregistered [ 892.924531] Key type ._llcrypt registered [ 892.927802] Key type .llcrypt registered [ 893.112474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 893.123584] alg: No test for adler32 (adler32-zlib) [ 894.044398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.142274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 894.196401] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 895.403410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.426360] LNet: 45309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.438604] LNetError: 45309:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.449789] LNet: Removed LNI 192.168.204.8@tcp [ 895.463285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.502929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 895.533895] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 902.969493] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 03:23:31 (1779693811) [ 903.099248] LNet: 45895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.108929] LNetError: 45895:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.150304] LNet: Removed LNI 192.168.204.8@tcp [ 903.952470] Key type .llcrypt unregistered [ 903.954645] Key type ._llcrypt unregistered [ 910.646703] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 910.735049] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 910.740098] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 910.749945] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 913.457787] Key type ._llcrypt registered [ 913.460122] Key type .llcrypt registered [ 913.692432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 913.712077] alg: No test for adler32 (adler32-zlib) [ 914.628618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.681490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 914.721872] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 915.659154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.687226] LNet: 46352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.699726] LNetError: 46352:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.715052] LNet: Removed LNI 192.168.204.8@tcp [ 915.731423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.811158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 915.850123] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 923.230300] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 03:23:52 (1779693832) [ 923.434907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 923.474200] LNet: 46949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.487672] LNetError: 46949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.524602] LNet: Removed LNI 192.168.204.8@tcp [ 923.573710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.716923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 930.208849] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 03:23:59 (1779693839) [ 930.402149] LNet: 47528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.976399] Key type .llcrypt unregistered [ 930.979064] Key type ._llcrypt unregistered [ 937.172921] Key type ._llcrypt registered [ 937.181362] Key type .llcrypt registered [ 937.544400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 937.579306] alg: No test for adler32 (adler32-zlib) [ 938.500626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.569481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 938.605671] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 939.400103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 939.419649] LNet: 47930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.423653] LNetError: 47930:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.438728] LNet: Removed LNI 192.168.204.8@tcp [ 939.465721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.514185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 939.531040] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 945.540253] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 03:24:14 (1779693854) [ 945.754650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.783364] LNet: 48523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.792156] LNetError: 48523:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.803595] LNet: Removed LNI 192.168.204.8@tcp [ 945.829948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.928466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 951.758536] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 03:24:20 (1779693860) [ 953.341791] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 955.504848] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 03:24:24 (1779693864) [ 957.293523] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 959.598680] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 03:24:27 (1779693867) [ 961.388134] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 963.184518] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 03:24:31 (1779693871) [ 964.598809] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 966.348788] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 03:24:35 (1779693875) [ 967.885221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 970.167203] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 03:24:38 (1779693878) [ 970.527161] LNet: 50575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.390628] Key type .llcrypt unregistered [ 971.392500] Key type ._llcrypt unregistered [ 977.919632] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 978.064202] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 978.067895] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 978.072682] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 980.753727] Key type ._llcrypt registered [ 980.760281] Key type .llcrypt registered [ 981.026952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 981.051243] alg: No test for adler32 (adler32-zlib) [ 982.124090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.189777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 982.219892] LNet: Added LNI 192.168.204.8@tcp [8/512/0/180] [ 982.226695] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 982.231190] LNet: 50890:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 982.323947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.340945] LNet: 50909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.346260] LNetError: 50909:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.351287] LNet: Removed LNI 192.168.204.8@tcp [ 982.354429] LNetError: 50909:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 982.367220] LNet: Removed LNI 10.1.2.3@tcp [ 982.390497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.476479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 982.506839] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 982.586161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.617659] LNet: 50933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.630562] LNetError: 50933:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.657864] LNet: Removed LNI 192.168.204.8@tcp [ 982.681789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.763556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 982.792968] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 982.806562] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 982.817189] LNet: 50941:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 982.955219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.993106] LNet: 50960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.002236] LNetError: 50960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.007673] LNet: Removed LNI 192.168.204.8@tcp [ 983.010303] LNetError: 50960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 983.025793] LNet: Removed LNI 10.1.2.3@tcp [ 983.053051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.155515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 983.175143] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 983.179735] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 983.181738] LNet: 50968:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 983.359601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.381731] LNet: 50994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.390993] LNetError: 50994:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.404311] LNet: Removed LNI 192.168.204.8@tcp [ 983.410801] LNetError: 50994:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 983.435078] LNet: Removed LNI 10.1.2.3@tcp [ 983.457068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.556925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 983.587389] LNet: Added LNI 192.168.204.8@tcp [8/256/0/182] [ 983.603817] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 983.759963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.790550] LNet: 51022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.799156] LNetError: 51022:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.803900] LNet: Removed LNI 192.168.204.8@tcp [ 983.807468] LNetError: 51022:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 983.821725] LNet: Removed LNI 10.1.2.3@tcp [ 983.845375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.969580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 984.004665] LNet: Added LNI 192.168.204.8@tcp [8/256/0/182] [ 984.014470] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 984.017584] LNet: 51030:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 984.129204] LNet: Removed LNI 192.168.204.8@tcp [ 984.145354] LNet: Removed LNI 10.1.2.3@tcp [ 984.787748] Key type .llcrypt unregistered [ 984.790471] Key type ._llcrypt unregistered [ 995.453978] Key type ._llcrypt registered [ 995.457158] Key type .llcrypt registered [ 996.703699] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 03:25:05 (1779693905) [ 997.365429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 997.382909] alg: No test for adler32 (adler32-zlib) [ 998.324577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.368856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 998.402875] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 999.049505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.8 [ 999.066494] LNet: 52005:0:(config.c:1628:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 999.073538] LNet: Removed LNI 192.168.204.8@tcp [ 1006.483617] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 03:25:15 (1779693915) [ 1006.608587] LNet: 52568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.621789] LNetError: 52568:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.649448] LNet: Removed LNI 192.168.204.8@tcp [ 1007.458858] Key type .llcrypt unregistered [ 1007.462838] Key type ._llcrypt unregistered [ 1013.876764] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1014.135350] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1017.032884] Key type ._llcrypt registered [ 1017.040983] Key type .llcrypt registered [ 1017.281325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1017.336057] alg: No test for adler32 (adler32-zlib) [ 1018.702709] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1026.111553] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 03:25:34 (1779693934) [ 1026.237378] LNet: 53480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.248365] LNetError: 53480:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1026.280524] LNet: Removed LNI 10.1.2.3@tcp [ 1027.058373] Key type .llcrypt unregistered [ 1027.060146] Key type ._llcrypt unregistered [ 1033.274049] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1033.496392] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1033.498340] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1033.507454] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1036.481816] Key type ._llcrypt registered [ 1036.485784] Key type .llcrypt registered [ 1036.616904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1036.630807] alg: No test for adler32 (adler32-zlib) [ 1037.676040] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1043.693400] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 03:25:52 (1779693952) [ 1044.960702] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1046.536614] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 03:25:55 (1779693955) [ 1046.728967] LNet: 54675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.739896] LNetError: 54675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1046.760208] LNet: Removed LNI 10.1.2.3@tcp [ 1047.235066] Key type .llcrypt unregistered [ 1047.238387] Key type ._llcrypt unregistered [ 1053.346128] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1053.599974] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1053.610628] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1053.619175] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1056.298255] Key type ._llcrypt registered [ 1056.301362] Key type .llcrypt registered [ 1056.495189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1056.522756] alg: No test for adler32 (adler32-zlib) [ 1057.404274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1057.476778] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1064.973550] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 03:26:13 (1779693973) [ 1065.387547] LNet: 55907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.392589] LNetError: 55907:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1065.417495] LNet: Removed LNI 10.1.2.3@tcp [ 1066.125066] Key type .llcrypt unregistered [ 1066.130852] Key type ._llcrypt unregistered [ 1071.277975] Key type ._llcrypt registered [ 1071.281888] Key type .llcrypt registered [ 1071.499719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1071.508627] alg: No test for adler32 (adler32-zlib) [ 1072.432237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1072.457263] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1080.021732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1080.884382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_interrupt -r 1 [ 1080.923955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_interrupt -r 1 [ 1080.967267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1080.986646] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1081.239150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1081.286164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1081.330282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1081.379308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1081.424656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1081.467693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1081.521263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1081.551241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1081.567357] LNet: 56344:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1081.589168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1081.618856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1081.651101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1081.683780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1081.695974] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1081.713422] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1081.721578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1081.750022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1081.810336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1082.424077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_dropped -r 1 [ 1082.477163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_dropped -r 1 [ 1082.526121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1082.806879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1082.835438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1082.923333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1082.946969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1082.983753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1083.029271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1083.076325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1083.104848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1083.144189] LNet: 56453:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1083.161775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1083.220452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1083.254220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1083.289650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1083.328595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1083.360695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1083.389746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1083.945825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_aborted -r 1 [ 1084.012327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_aborted -r 1 [ 1084.063667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1084.087818] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1084.102460] LNet: Skipped 1 previous similar message [ 1084.313655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1084.367289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1084.411592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1084.439986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1084.470624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1084.496686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1084.528197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1084.557819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1084.580179] LNet: 56563:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1084.607917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1084.661408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1084.706996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1084.737807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1084.786111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1084.829313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1084.869583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1085.400731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_no_route -r 1 [ 1085.436184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_no_route -r 1 [ 1085.472495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1085.816656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1085.869425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1085.904211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1085.955192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1085.994769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1086.030222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1086.072548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1086.101399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1086.131192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1086.170461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1086.201069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1086.233073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1086.262355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1086.294574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1086.333799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1086.911943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_timeout -r 1 [ 1086.947063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_timeout -r 1 [ 1086.975451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1086.992873] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1086.997990] LNet: Skipped 1 previous similar message [ 1087.239186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1087.286393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1087.338331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1087.387480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1087.444348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1087.493138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1087.550731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1087.606590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1087.653197] LNet: 56781:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1087.658603] LNet: 56781:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1087.680906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1087.719667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1087.742765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1087.777782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1087.834185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1087.878069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1087.959957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1088.520941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_error -r 1 [ 1088.558767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_error -r 1 [ 1088.616072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1088.834635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1088.873964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1088.920701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1088.974873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1089.041071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1089.096647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1089.168868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1089.213219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1089.266307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1089.340950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1089.387957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1089.438300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1089.492572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1089.564632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1089.634965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1089.956541] LNet: 56932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.964308] LNetError: 56932:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1090.988623] LNet: Removed LNI 192.168.204.8@tcp [ 1091.507890] Key type .llcrypt unregistered [ 1091.511409] Key type ._llcrypt unregistered [ 1110.863935] Key type ._llcrypt registered [ 1110.871236] Key type .llcrypt registered [ 1112.225246] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 03:27:00 (1779694020) [ 1113.141688] Key type .llcrypt unregistered [ 1113.148174] Key type ._llcrypt unregistered [ 1119.086554] Key type ._llcrypt registered [ 1119.093700] Key type .llcrypt registered [ 1119.300090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1119.317720] alg: No test for adler32 (adler32-zlib) [ 1120.362435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1120.423321] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1128.087894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1138.063118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1138.120759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1138.151395] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1139.474306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_interrupt -r 1 [ 1139.518855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_interrupt -r 1 [ 1139.566752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_interrupt -r 1 [ 1139.597646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_interrupt -r 1 [ 1139.630812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_interrupt -r 1 [ 1139.662022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_interrupt -r 1 [ 1139.709817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_interrupt -r 1 [ 1139.753154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_interrupt -r 1 [ 1139.834380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1139.873177] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1139.881889] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1140.176416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.215681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1140.249858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1140.278383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1140.317685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.382074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1140.430918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1140.500348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1140.520272] LNet: 58401:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1140.563621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1140.594457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1140.634362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1140.668891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.695091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1140.730431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1140.763415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1141.405172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_dropped -r 1 [ 1141.458569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_dropped -r 1 [ 1141.493340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_dropped -r 1 [ 1141.541340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_dropped -r 1 [ 1141.584900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_dropped -r 1 [ 1141.641361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_dropped -r 1 [ 1141.709958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_dropped -r 1 [ 1141.751900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_dropped -r 1 [ 1141.797766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1141.827623] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1141.831991] LNet: Skipped 2 previous similar messages [ 1142.029577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1142.060926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1142.086555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1142.107953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1142.134571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1142.158902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1142.196621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1142.235146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1142.251383] LNet: 58523:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1142.267228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1142.293688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1142.321799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1142.353874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1142.406974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1142.468744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1142.521594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1143.200249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_aborted -r 1 [ 1143.254045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_aborted -r 1 [ 1143.309162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_aborted -r 1 [ 1143.348774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_aborted -r 1 [ 1143.420239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_aborted -r 1 [ 1143.503428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_aborted -r 1 [ 1143.563278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_aborted -r 1 [ 1143.620121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_aborted -r 1 [ 1143.680993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1143.727459] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1143.735421] LNet: Skipped 2 previous similar messages [ 1144.036679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1144.086043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1144.141108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1144.204711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1144.251356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1144.286631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1144.318090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1144.387487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1144.419667] LNet: 58644:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1144.457695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1144.488522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1144.547833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1144.582628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1144.624495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1144.674580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1144.746118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1145.372563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_no_route -r 1 [ 1145.406562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_no_route -r 1 [ 1145.438906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_no_route -r 1 [ 1145.468494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_no_route -r 1 [ 1145.499851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_no_route -r 1 [ 1145.538507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_no_route -r 1 [ 1145.590415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_no_route -r 1 [ 1145.653983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_no_route -r 1 [ 1145.710718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1145.748222] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1145.756811] LNet: Skipped 2 previous similar messages [ 1146.023873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1146.087491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1146.144644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1146.215256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1146.273989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1146.315380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1146.352745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1146.391647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1146.431786] LNet: 58766:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1146.463431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1146.516729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1146.575208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1146.620769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1146.676585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1146.734153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1146.819120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1147.455279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_timeout -r 1 [ 1147.493955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_timeout -r 1 [ 1147.523631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_timeout -r 1 [ 1147.556251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_timeout -r 1 [ 1147.586029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_timeout -r 1 [ 1147.618848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_timeout -r 1 [ 1147.660411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_timeout -r 1 [ 1147.706260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_timeout -r 1 [ 1147.753954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1148.028595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1148.093109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1148.149320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1148.204560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1148.268700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1148.334617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1148.370651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1148.404560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1148.435345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1148.485314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1148.522644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1148.561788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1148.606614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1148.659683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1148.705716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1149.279477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_error -r 1 [ 1149.338730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_error -r 1 [ 1149.378720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_error -r 1 [ 1149.415838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_error -r 1 [ 1149.441748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_error -r 1 [ 1149.471900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_error -r 1 [ 1149.514683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_error -r 1 [ 1149.553610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_error -r 1 [ 1149.586595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1149.845688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1149.878537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1149.917129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1149.964657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1150.027754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1150.073793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1150.121940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1150.157759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1150.185137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1150.224901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1150.268495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1150.309114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1150.354836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1150.413494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1150.473695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1156.073516] LNet: 59084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.079150] LNetError: 59084:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.087812] LNet: Removed LNI 192.168.204.8@tcp [ 1157.162948] LNet: Removed LNI 192.168.204.8@tcp1 [ 1157.552780] Key type .llcrypt unregistered [ 1157.557319] Key type ._llcrypt unregistered [ 1175.254229] Key type ._llcrypt registered [ 1175.258366] Key type .llcrypt registered [ 1176.795808] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 03:28:05 (1779694085) [ 1177.851418] Key type .llcrypt unregistered [ 1177.853673] Key type ._llcrypt unregistered [ 1185.304900] Key type ._llcrypt registered [ 1185.317894] Key type .llcrypt registered [ 1185.641991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1185.656939] alg: No test for adler32 (adler32-zlib) [ 1186.660331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1186.688098] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1194.410741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1195.234191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e remote_dropped -r 1 [ 1195.286425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e remote_dropped -r 1 [ 1195.336681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1195.357800] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1195.739367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1195.776499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1195.857585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1195.945981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1195.994829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1196.040589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1196.081257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1196.119729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1196.138183] LNet: 60337:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1196.162870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1196.232319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1196.295549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1196.382282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1196.454307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1196.517324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1196.584300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1197.230852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e remote_error -r 1 [ 1197.314862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e remote_error -r 1 [ 1197.429972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1197.480108] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1197.821801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1197.854986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1197.886736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1197.923563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1197.958635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1197.996901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1198.020616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1198.087912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1198.120761] LNet: 60447:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1198.142826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1198.173650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1198.216354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1198.252621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1198.295584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1198.342579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1198.391344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1199.206901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e remote_timeout -r 1 [ 1199.261664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e remote_timeout -r 1 [ 1199.311028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1199.338165] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1199.638470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1199.698365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1199.748790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1199.777437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1199.804610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1199.833504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1199.855525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1199.920624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1199.966354] LNet: 60556:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1199.987146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1200.026691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1200.073596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1200.114568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1200.150992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1200.202911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1200.260386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1200.663180] LNet: 60597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.677496] LNetError: 60597:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1200.707140] LNet: Removed LNI 192.168.204.8@tcp [ 1201.138719] Key type .llcrypt unregistered [ 1201.142188] Key type ._llcrypt unregistered [ 1216.719419] Key type ._llcrypt registered [ 1216.723263] Key type .llcrypt registered [ 1218.318780] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 03:28:46 (1779694126) [ 1219.297508] Key type .llcrypt unregistered [ 1219.301093] Key type ._llcrypt unregistered [ 1225.870128] Key type ._llcrypt registered [ 1225.873366] Key type .llcrypt registered [ 1226.253439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1226.274993] alg: No test for adler32 (adler32-zlib) [ 1227.201445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1227.232652] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1234.599814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1243.666952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1243.714651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1243.744548] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1245.676048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e remote_dropped -r 1 [ 1245.774362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e remote_dropped -r 1 [ 1245.837199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e remote_dropped -r 1 [ 1245.901545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e remote_dropped -r 1 [ 1245.994295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e remote_dropped -r 1 [ 1246.100990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e remote_dropped -r 1 [ 1246.174948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e remote_dropped -r 1 [ 1246.276704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e remote_dropped -r 1 [ 1246.384737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1246.434321] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1246.442644] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.108@tcp [ 1246.791834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1246.828759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1246.887915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1246.923335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1246.968681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1247.022809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1247.062678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.095499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.125598] LNet: 62066:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1247.153288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1247.192843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1247.238679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1247.283391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1247.343277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1247.399963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1247.444556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1247.487272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.093838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e remote_error -r 1 [ 1248.165958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e remote_error -r 1 [ 1248.217541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e remote_error -r 1 [ 1248.274243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e remote_error -r 1 [ 1248.340980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e remote_error -r 1 [ 1248.400756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e remote_error -r 1 [ 1248.453161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e remote_error -r 1 [ 1248.505629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e remote_error -r 1 [ 1248.541576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1248.561672] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1248.565802] LNet: Skipped 2 previous similar messages [ 1248.785245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1248.826371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1248.858924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1248.888748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1248.917747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1248.959155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.002442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.046050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.077404] LNet: 62189:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1249.101567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.148370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.193481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.219250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.247359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.286985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1249.327541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1249.365592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.977976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e remote_timeout -r 1 [ 1250.036251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e remote_timeout -r 1 [ 1250.091664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e remote_timeout -r 1 [ 1250.131973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e remote_timeout -r 1 [ 1250.186652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e remote_timeout -r 1 [ 1250.231520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e remote_timeout -r 1 [ 1250.293892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e remote_timeout -r 1 [ 1250.345408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e remote_timeout -r 1 [ 1250.374198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.108@tcp [ 1250.391515] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1250.661719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1250.695511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1250.744232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1250.785917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1250.856413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1250.930441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1250.967213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1251.009416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1251.053235] LNet: 62313:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1251.084343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1251.129757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1251.202933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1251.256209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1251.302144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1251.337535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1251.374198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1251.415215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1255.617280] LNet: 62391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1255.621947] LNetError: 62391:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1255.632570] LNet: Removed LNI 192.168.204.8@tcp [ 1255.648265] LNet: Removed LNI 192.168.204.8@tcp1 [ 1256.152875] Key type .llcrypt unregistered [ 1256.155512] Key type ._llcrypt unregistered [ 1272.942274] Key type ._llcrypt registered [ 1272.946684] Key type .llcrypt registered [ 1274.701746] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 03:29:42 (1779694182) [ 1276.592248] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1278.502913] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 03:29:47 (1779694187) [ 1279.397556] Key type .llcrypt unregistered [ 1279.402395] Key type ._llcrypt unregistered [ 1285.701163] Key type ._llcrypt registered [ 1285.704402] Key type .llcrypt registered [ 1286.019831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1286.053938] alg: No test for adler32 (adler32-zlib) [ 1286.989053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1287.030581] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1295.021367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1295.949958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e network_timeout -r 1 [ 1296.007198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e network_timeout -r 1 [ 1296.068917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1296.095109] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1296.287835] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1296.359085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1296.400761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1296.432850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1296.466777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1296.499158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1296.530543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1296.581533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1296.629453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1296.646187] LNet: 63936:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1296.662289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1296.701606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1296.737345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1296.767903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1296.810687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.860307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1297.168892] LNet: 63975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.181757] LNetError: 63975:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1297.201653] LNet: Removed LNI 192.168.204.8@tcp [ 1297.656464] Key type .llcrypt unregistered [ 1297.659254] Key type ._llcrypt unregistered [ 1316.895908] Key type ._llcrypt registered [ 1316.897462] Key type .llcrypt registered [ 1317.103306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1317.129868] alg: No test for adler32 (adler32-zlib) [ 1318.138025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1318.192147] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1326.139462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1335.345305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1335.417935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1335.445566] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1337.435620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e network_timeout -r 1 [ 1337.523253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e network_timeout -r 1 [ 1337.591210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e network_timeout -r 1 [ 1337.648249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e network_timeout -r 1 [ 1337.685894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e network_timeout -r 1 [ 1337.753532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e network_timeout -r 1 [ 1337.826981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e network_timeout -r 1 [ 1337.875121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e network_timeout -r 1 [ 1337.902723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1337.939954] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1338.222151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1338.254844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1338.294422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1338.362700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1338.423902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1338.461667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1338.499355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1338.542907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1338.559458] LNet: 64856:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1338.577371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1338.604411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1338.632185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1338.667983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1338.697251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1338.721748] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1338.726652] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1338.738036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1343.153727] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1343.477598] LNet: 64930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1343.488886] LNetError: 64930:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.498373] LNet: Removed LNI 192.168.204.8@tcp [ 1343.513818] LNet: Removed LNI 192.168.204.8@tcp1 [ 1344.024513] Key type .llcrypt unregistered [ 1344.034295] Key type ._llcrypt unregistered [ 1359.759025] Key type ._llcrypt registered [ 1359.762100] Key type .llcrypt registered [ 1361.307759] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 03:31:09 (1779694269) [ 1361.746204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1361.760448] alg: No test for adler32 (adler32-zlib) [ 1362.756358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1362.837557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1362.882096] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1363.692491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1363.711553] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1364.823046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 1364.908905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1364.976205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1365.015669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1365.053484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1365.094505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1365.130280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.8@tcp [ 1365.154613] LNetError: 66031:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.8@tcp to recovery [ 1365.855853] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1365.855980] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1367.903433] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1368.760300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1371.999725] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1372.500392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1372.553569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1372.638960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1372.674650] LNet: 66139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1372.682355] LNetError: 66139:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1372.694145] LNet: Removed LNI 192.168.204.8@tcp [ 1372.711677] LNet: Removed LNI 192.168.204.8@tcp1 [ 1372.744024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1372.798122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1372.827520] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1373.735305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1373.767705] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1374.671865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 1374.724625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1374.797637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1374.864605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1374.913926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1374.979338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1375.049947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1375.110872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.8@tcp [ 1375.133512] LNetError: 66411:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.8@tcp to recovery [ 1375.839980] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1378.715611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1381.987481] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1381.999039] LNet: Skipped 1 previous similar message [ 1385.661759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1385.717441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1385.755582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1394.303599] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 03:31:42 (1779694302) [ 1394.440871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1394.497650] LNet: 67111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1394.519318] LNetError: 67111:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1394.533920] LNetError: 67111:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1394.547298] LNet: Removed LNI 192.168.204.8@tcp [ 1394.570314] LNet: Removed LNI 192.168.204.8@tcp1 [ 1394.633987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1394.713889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1394.751351] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1395.851570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1395.875320] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1396.986406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 1397.062138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1397.100491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1397.141052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1397.194567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1397.244378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1397.290304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.8@tcp --health 0 [ 1397.392698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1397.727348] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.8@tcp [ 1398.752192] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1398.761893] LNet: Skipped 1 previous similar message [ 1412.306319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1412.591899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1412.651596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1412.733711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.8@tcp --health 500 [ 1412.904640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1418.731504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1418.862458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1418.905139] LNet: 67594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1418.917487] LNetError: 67594:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1418.932471] LNetError: 67594:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1418.938836] LNet: Removed LNI 192.168.204.8@tcp [ 1418.959608] LNet: Removed LNI 192.168.204.8@tcp1 [ 1419.013236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1419.130862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1419.185262] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1420.077660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1420.100034] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1421.139801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 1421.175632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1421.234834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1421.259449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1421.305410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1421.358789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1421.402444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1421.433741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.8@tcp --health 0 [ 1422.112563] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1422.128430] LNet: Skipped 3 previous similar messages [ 1422.767696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1433.295813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1433.336622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1433.382203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1442.846178] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 03:32:30 (1779694350) [ 1443.678926] LNet: 68644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1443.688742] LNetError: 68644:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1443.697700] LNetError: 68644:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1443.708289] LNet: Removed LNI 192.168.204.8@tcp [ 1443.731276] LNet: Removed LNI 192.168.204.8@tcp1 [ 1444.864208] Key type .llcrypt unregistered [ 1444.872139] Key type ._llcrypt unregistered [ 1451.814489] Key type ._llcrypt registered [ 1451.819069] Key type .llcrypt registered [ 1452.081908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1452.113942] alg: No test for adler32 (adler32-zlib) [ 1453.139264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1453.181042] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1463.258681] Lustre: DEBUG MARKER: Initial discovery [ 1463.329238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.108@tcp [ 1465.993446] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1466.036679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1466.098207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.108@tcp [ 1466.136619] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1466.153842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1466.198347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.108@tcp -r 1 -m GET -l 3 [ 1466.243693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.108@tcp -r 1 -m GET -e local_error [ 1466.305447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.108@tcp -r 1 -m PUT -l 6 [ 1466.591269] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1467.761515] Lustre: DEBUG MARKER: Force oleg408-client.virtnet to discover 192.168.204.108@tcp (in background) [ 1468.101467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.108@tcp [ 1469.588665] Lustre: DEBUG MARKER: Force oleg408-server to discover 192.168.204.8@tcp [ 1471.203561] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1472.103277] Lustre: DEBUG MARKER: Wait for 69411 [ 1473.294239] Lustre: DEBUG MARKER: Finished wait on 69411 [ 1473.529488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1473.567286] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1473.574694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1473.867903] LNet: 69860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1473.880230] LNetError: 69860:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1473.899507] LNet: Removed LNI 192.168.204.8@tcp [ 1474.195925] Key type .llcrypt unregistered [ 1474.198791] Key type ._llcrypt unregistered [ 1489.003748] Key type ._llcrypt registered [ 1489.009592] Key type .llcrypt registered [ 1490.308595] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 03:33:19 (1779694399) [ 1490.661298] Key type .llcrypt unregistered [ 1490.666439] Key type ._llcrypt unregistered [ 1496.393471] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1496.505809] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1496.514601] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1496.525505] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1499.231553] Key type ._llcrypt registered [ 1499.234833] Key type .llcrypt registered [ 1499.488933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1499.508381] alg: No test for adler32 (adler32-zlib) [ 1500.460895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1500.511907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1500.554693] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1501.141553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1501.172501] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1510.355797] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 03:33:38 (1779694418) [ 1510.498026] LNet: 71930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1510.509077] LNetError: 71930:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1510.517703] LNet: Removed LNI 192.168.204.8@tcp [ 1510.540001] LNet: Removed LNI 10.1.2.3@tcp [ 1511.309539] Key type .llcrypt unregistered [ 1511.313279] Key type ._llcrypt unregistered [ 1518.159674] Key type ._llcrypt registered [ 1518.168656] Key type .llcrypt registered [ 1518.486885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1518.506577] alg: No test for adler32 (adler32-zlib) [ 1519.461419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.506930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1519.524505] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1520.335465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1520.364569] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1521.471842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.8@tcp [ 1522.528679] LNetError: Unexpected error -22 connecting to 192.168.204.8@tcp at host 192.168.204.8:988 [ 1529.809137] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 03:33:58 (1779694438) [ 1529.922160] LNet: 73063:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1529.930533] LNetError: 73063:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1529.940426] LNet: Removed LNI 192.168.204.8@tcp [ 1529.972418] LNet: Removed LNI 10.1.2.3@tcp [ 1530.840446] Key type .llcrypt unregistered [ 1530.843197] Key type ._llcrypt unregistered [ 1537.554424] Key type ._llcrypt registered [ 1537.563273] Key type .llcrypt registered [ 1537.863235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1537.900257] alg: No test for adler32 (adler32-zlib) [ 1539.015105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.139351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1539.204981] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1539.755533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1539.774389] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 1540.764621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.8@tcp1 --nid 192.168.204.8@tcp2 [ 1545.428905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp1 192.168.204.8@tcp2 [ 1545.454470] LNetError: 73907:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.8@tcp1 to 192.168.204.8@tcp2 - Route table may be misconfigured [ 1545.475914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp2 192.168.204.8@tcp1 [ 1553.244792] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 03:34:22 (1779694462) [ 1553.543520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1553.583822] LNet: 74482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.609545] LNetError: 74482:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.657704] LNet: Removed LNI 192.168.204.8@tcp1 [ 1554.660825] LNetError: 74482:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.697060] LNet: Removed LNI 192.168.204.8@tcp2 [ 1554.761103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.872932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1554.894822] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1555.744810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1555.774439] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1556.674760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 1556.714601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp 192.168.204.8@tcp [ 1556.743487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp1 192.168.204.8@tcp1 [ 1556.771616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.8@tcp -d 192.168.204.8@tcp -e network_timeout [ 1556.798523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -e network_timeout [ 1556.827450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -e network_timeout [ 1556.861300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -e network_timeout [ 1556.892455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1556.910898] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1556.930530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1557.017695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1557.107535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1557.920205] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1563.837679] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 03:34:32 (1779694472) [ 1564.136726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.157808] LNet: 75344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1564.164649] LNetError: 75344:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1564.171367] LNet: Removed LNI 192.168.204.8@tcp [ 1565.226746] LNet: Removed LNI 192.168.204.8@tcp1 [ 1565.278155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.406771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1565.698843] LNet: 75381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1566.186148] Key type .llcrypt unregistered [ 1566.195186] Key type ._llcrypt unregistered [ 1578.501761] Key type ._llcrypt registered [ 1578.511728] Key type .llcrypt registered [ 1580.152434] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 03:34:48 (1779694488) [ 1580.603085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1580.628721] alg: No test for adler32 (adler32-zlib) [ 1581.591861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.162261] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1585.335690] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 03:34:53 (1779694493) [ 1585.418807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.446289] LNet: 76474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.471992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.548756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1585.596797] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1586.603919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1586.661826] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1587.840851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1587.913818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp1 [ 1587.982417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp1 [ 1595.219285] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 03:35:03 (1779694503) [ 1598.009741] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg408-server [ 1600.060578] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 03:35:08 (1779694508) [ 1602.037968] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg408-server [ 1603.700452] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 03:35:12 (1779694512) [ 1606.108521] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg408-server [ 1607.888674] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 03:35:16 (1779694516) [ 1610.204696] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg408-server [ 1612.033334] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 03:35:20 (1779694520) [ 1614.357451] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg408-server [ 1616.459161] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 03:35:24 (1779694524) [ 1619.148617] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg408-server [ 1621.277721] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 03:35:29 (1779694529) [ 1623.302970] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg408-server [ 1625.268664] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 03:35:33 (1779694533) [ 1627.466742] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg408-server [ 1629.235575] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 03:35:38 (1779694538) [ 1631.316392] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg408-server [ 1633.518395] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 03:35:42 (1779694542) [ 1633.659668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1633.693930] LNet: 80356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.709380] LNetError: 80356:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.717614] LNet: Removed LNI 192.168.204.8@tcp [ 1633.748873] LNet: Removed LNI 192.168.204.8@tcp1 [ 1633.805511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.854852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1633.883063] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1634.894979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1634.981526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1635.255586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.291282] LNet: 80507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1635.297245] LNetError: 80507:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1635.302673] LNetError: 80507:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1635.316171] LNet: Removed LNI 192.168.204.8@tcp [ 1635.342809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.449478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1635.486231] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1636.712932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1636.989882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.026521] LNetError: 80652:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1638.057747] LNet: Removed LNI 192.168.204.8@tcp [ 1638.110560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.245654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1638.283674] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1639.024576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1639.154314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1639.430644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.477685] LNet: 80800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.480877] LNet: 80800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1639.497423] LNetError: 80800:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1640.551167] LNet: Removed LNI 192.168.204.8@tcp [ 1640.600259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.666391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1640.700546] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1641.720786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1641.914977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1643.000136] LNet: Removed LNI 192.168.204.8@tcp [ 1643.040463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1643.106388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1643.140574] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1644.121768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1644.256849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1645.836057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.868817] LNet: 81101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.873136] LNet: 81101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1645.878508] LNetError: 81101:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1645.882817] LNetError: 81101:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1645.907574] LNet: Removed LNI 192.168.204.8@tcp [ 1645.934706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.008647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1646.065512] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1647.133592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1647.512523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.615381] LNet: Removed LNI 192.168.204.8@tcp [ 1648.673123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.759672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1648.802914] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1649.753330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1649.880127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1650.168840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1651.242769] LNet: Removed LNI 192.168.204.8@tcp [ 1651.297327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.402795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1651.442314] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1652.309496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1652.570518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.614316] LNet: Removed LNI 192.168.204.8@tcp [ 1653.662406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.737174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1653.782062] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1654.882865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1654.982648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1655.295606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.330751] LNet: 81687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.338736] LNet: 81687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1655.348032] LNetError: 81687:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.354939] LNetError: 81687:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1656.426498] LNet: Removed LNI 192.168.204.8@tcp [ 1656.494278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.574562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1656.617189] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1657.578077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1657.973473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1659.045904] LNet: Removed LNI 192.168.204.8@tcp [ 1659.117456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.237346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1659.291551] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1660.543490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1660.812766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1661.317762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.445145] LNet: Removed LNI 192.168.204.8@tcp [ 1662.506502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.629361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1662.725343] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1663.973334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1667.120073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1668.202093] LNet: Removed LNI 192.168.204.8@tcp [ 1668.280573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.403172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1668.431700] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1669.321428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1669.409516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1669.450416] LNet: Removed LNI 192.168.204.8@tcp [ 1669.479836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.544865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1669.574764] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1670.372252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1677.703199] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 03:36:26 (1779694586) [ 1677.854437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1677.879576] LNet: 82979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1677.891761] LNet: 82979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1677.909690] LNetError: 82979:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1677.924712] LNetError: 82979:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1677.951205] LNet: Removed LNI 192.168.204.8@tcp [ 1677.973988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.050926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1678.083036] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/180] [ 1679.384303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1679.437799] LNet: Removed LNI 192.168.204.8@tcp231 [ 1679.465688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.566305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1679.604581] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/180] [ 1679.616152] LNet: 83128:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1680.027107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1680.060115] LNet: Removed LNI 192.168.204.8@tcp231 [ 1680.081195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1680.124967] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/180] [ 1681.368336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1681.429430] LNet: Removed LNI 192.168.204.8@tcp231 [ 1681.475757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.588204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1681.652133] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/0] [ 1681.660939] LNet: 83280:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1682.120625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1682.153736] LNet: Removed LNI 192.168.204.8@tcp231 [ 1682.188839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1682.220593] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/0] [ 1683.601158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.660226] LNet: Removed LNI 192.168.204.8@tcp231 [ 1683.679247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.739752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1683.781478] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/60] [ 1683.795279] LNet: 83432:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1684.322772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1684.370376] LNet: Removed LNI 192.168.204.8@tcp231 [ 1684.387540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1684.426186] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/60] [ 1685.824508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.863298] LNet: Removed LNI 192.168.204.8@tcp231 [ 1685.904967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.996458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1686.031650] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/180] [ 1686.038581] LNet: 83583:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1686.716442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1686.754929] LNet: Removed LNI 192.168.204.8@tcp231 [ 1686.774097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1686.810723] LNet: Added LNI 192.168.204.8@tcp231 [8/256/0/180] [ 1696.024352] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 03:36:44 (1779694604) [ 1696.121420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.197537] LNet: Removed LNI 192.168.204.8@tcp231 [ 1696.227371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.319911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1696.350977] LNet: Added LNI 192.168.204.8@tcp232 [8/256/0/180] [ 1697.114764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1697.166131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1697.202799] LNet: Removed LNI 192.168.204.8@tcp232 [ 1697.219884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1697.244254] LNet: Added LNI 192.168.204.8@tcp232 [8/256/0/180] [ 1697.887145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1697.954360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1697.999835] LNet: Removed LNI 192.168.204.8@tcp232 [ 1698.012050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1698.032421] LNet: Added LNI 192.168.204.8@tcp232 [8/256/0/180] [ 1698.836364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1698.880585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1698.918629] LNet: Removed LNI 192.168.204.8@tcp232 [ 1698.942637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1698.990236] LNet: Added LNI 192.168.204.8@tcp232 [8/256/0/180] [ 1699.897717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1699.938169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1699.960288] LNet: Removed LNI 192.168.204.8@tcp232 [ 1699.979906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1706.897736] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 03:36:55 (1779694615) [ 1708.293156] Key type .llcrypt unregistered [ 1708.294863] Key type ._llcrypt unregistered [ 1715.187748] Key type ._llcrypt registered [ 1715.194398] Key type .llcrypt registered [ 1715.441274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1715.455971] alg: No test for adler32 (adler32-zlib) [ 1716.405776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1716.446628] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1724.564706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1734.526753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.597846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1734.621022] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1736.670932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_interrupt -r 2 [ 1736.721187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_interrupt -r 2 [ 1736.840787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_interrupt -r 2 [ 1736.915153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_interrupt -r 2 [ 1736.966049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_interrupt -r 2 [ 1737.033977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_interrupt -r 2 [ 1737.064518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_interrupt -r 2 [ 1737.118872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_interrupt -r 2 [ 1737.182964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp 192.168.204.108@tcp [ 1737.232434] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1737.252607] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1737.699661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1737.738485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1737.795680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1737.829548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1737.869250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1737.911122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1737.943905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1737.973844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1737.994795] LNet: 86085:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1738.018219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1738.060944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1738.086527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1738.113313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1738.144208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1738.170836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1738.809901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_dropped -r 2 [ 1738.845134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_dropped -r 2 [ 1738.895656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_dropped -r 2 [ 1738.939542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_dropped -r 2 [ 1738.985818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_dropped -r 2 [ 1739.042961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_dropped -r 2 [ 1739.097877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_dropped -r 2 [ 1739.157334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_dropped -r 2 [ 1739.232297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp 192.168.204.108@tcp [ 1739.267815] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1739.669343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1739.712916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1739.781801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1739.840622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1739.906913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1739.962818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1740.027902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1740.082797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1740.117286] LNet: 86205:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1740.168482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1740.236170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1740.289280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1740.329554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1740.367451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1740.404335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1741.048982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_aborted -r 2 [ 1741.117648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_aborted -r 2 [ 1741.178451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_aborted -r 2 [ 1741.254843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_aborted -r 2 [ 1741.321848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_aborted -r 2 [ 1741.383648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_aborted -r 2 [ 1741.442802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_aborted -r 2 [ 1741.494112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_aborted -r 2 [ 1741.535503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp 192.168.204.108@tcp [ 1741.573737] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1741.904597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1741.945326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1741.988939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1742.040652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1742.083949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1742.114456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1742.153411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1742.195168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1742.235651] LNet: 86325:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1742.261649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1742.318724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1742.366844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1742.393864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1742.419978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1742.458062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1742.997566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_no_route -r 2 [ 1743.041338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_no_route -r 2 [ 1743.080589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_no_route -r 2 [ 1743.122323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_no_route -r 2 [ 1743.191929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_no_route -r 2 [ 1743.258378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_no_route -r 2 [ 1743.321862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_no_route -r 2 [ 1743.382584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_no_route -r 2 [ 1743.440810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp 192.168.204.108@tcp [ 1743.782718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1743.837614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1743.892408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1743.945269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1744.029725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1744.087961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1744.128541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1744.172796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1744.227435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1744.277904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1744.345825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1744.396892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1744.467831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1744.551089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1745.271696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e local_timeout -r 2 [ 1745.343575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e local_timeout -r 2 [ 1745.399425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e local_timeout -r 2 [ 1745.451520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e local_timeout -r 2 [ 1745.505164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e local_timeout -r 2 [ 1745.609372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e local_timeout -r 2 [ 1745.670520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e local_timeout -r 2 [ 1745.715698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e local_timeout -r 2 [ 1745.761037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp 192.168.204.108@tcp [ 1745.797585] LNet: There was an unexpected network error while writing to 192.168.204.108: rc = -22 [ 1745.803708] LNet: Skipped 1 previous similar message [ 1746.170057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1746.226892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1746.289362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1746.333377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1746.375247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1746.421796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1746.472388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1746.508898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1746.539502] LNet: 86566:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1746.552146] LNet: 86566:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1746.575057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1746.624155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1746.669970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1746.697311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1746.737742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1746.781871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1747.430601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp -m GET -e remote_dropped -r 2 [ 1747.498253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.108@tcp1 -m GET -e remote_dropped -r 2 [ 1747.539486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp -m GET -e remote_dropped -r 2 [ 1747.579252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp -d 192.168.204.8@tcp1 -m GET -e remote_dropped -r 2 [ 1747.625613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp -m GET -e remote_dropped -r 2 [ 1747.663298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.108@tcp1 -m GET -e remote_dropped -r 2 [ 1747.708522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp -m GET -e remote_dropped -r 2 [ 1747.757780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.8@tcp1 -d 192.168.204.8@tcp1 -m GET -e remote_dropped -r 2 [ 1747.802544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.8@tcp 192.168.204.108@tcp [ 1747.840269] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.108@tcp [ 1748.100067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1748.134936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1748.195115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1748.250671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1748.323137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1748.383153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1748.418671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1748.468708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1748.504190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1748.547466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1748.581946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1748.646586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1748.685586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1748.732910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1755.332381] LNet: 86760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.336687] LNetError: 86760:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.346392] LNet: Removed LNI 192.168.204.8@tcp [ 1755.361963] LNet: Removed LNI 192.168.204.8@tcp1 [ 1755.967773] Key type .llcrypt unregistered [ 1755.973763] Key type ._llcrypt unregistered [ 1776.906767] Key type ._llcrypt registered [ 1776.910232] Key type .llcrypt registered [ 1778.885047] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 03:38:06 (1779694686) [ 1779.578603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1779.637627] alg: No test for adler32 (adler32-zlib) [ 1780.646805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.731483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1780.776629] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1781.766614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1781.814707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.8@tcp [ 1781.853382] LNetError: 87738:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.8@tcp to recovery [ 1782.006703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.8@tcp [ 1782.154156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1790.423320] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 03:38:18 (1779694698) [ 1790.565522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1790.594753] LNet: 88323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1790.602634] LNetError: 88323:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1790.639356] LNet: Removed LNI 192.168.204.8@tcp [ 1790.685288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1790.828922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1790.874605] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1791.876630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1791.898163] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 1793.083873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.8@tcp [ 1793.111674] LNetError: 88594:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.8@tcp to recovery [ 1793.896386] LNet: 1 local NIs in recovery (showing 1): 192.168.204.8@tcp [ 1793.898214] LNet: There was an unexpected network error while writing to 192.168.204.8: rc = -22 [ 1794.468024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.8@tcp [ 1805.061948] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 03:38:33 (1779694713) [ 1806.933243] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg408-server [ 1809.027532] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 03:38:37 (1779694717) [ 1811.318307] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg408-server [ 1813.225582] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 03:38:41 (1779694721) [ 1815.710525] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg408-server [ 1817.564864] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 03:38:46 (1779694726) [ 1817.768562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1817.810976] LNet: 90224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1817.823538] LNetError: 90224:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1817.836136] LNet: Removed LNI 192.168.204.8@tcp [ 1817.872351] LNet: Removed LNI 192.168.204.8@tcp1 [ 1817.893427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1817.958601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1817.997191] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1818.905829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1818.995104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1819.072989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1819.179813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1819.218674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1819.275592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1826.682799] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 03:38:55 (1779694735) [ 1826.852341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.886223] LNet: 90949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1826.897589] LNetError: 90949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1826.906954] LNetError: 90949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1826.942969] LNet: Removed LNI 192.168.204.8@tcp [ 1826.961235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.015210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1827.047285] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1836.051205] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 03:39:04 (1779694744) [ 1836.195505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1836.228747] LNet: 91651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1836.239140] LNetError: 91651:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1836.270394] LNet: Removed LNI 192.168.204.8@tcp [ 1836.314685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.377592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1836.402782] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1844.530996] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 03:39:12 (1779694752) [ 1846.642638] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1848.891108] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 03:39:17 (1779694757) [ 1849.515054] LNet: 92690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.522310] LNetError: 92690:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1849.532807] LNet: Removed LNI 192.168.204.8@tcp [ 1850.470371] Key type .llcrypt unregistered [ 1850.476503] Key type ._llcrypt unregistered [ 1857.595159] Key type ._llcrypt registered [ 1857.596990] Key type .llcrypt registered [ 1857.824409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1857.833656] alg: No test for adler32 (adler32-zlib) [ 1858.831412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1858.870311] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1868.080227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 1874.783224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.108@tcp [ 1875.319232] LNet: 93108:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1875.326379] LNetError: 93108:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1875.349348] LNet: Removed LNI 192.168.204.8@tcp [ 1875.896580] Key type .llcrypt unregistered [ 1875.900288] Key type ._llcrypt unregistered [ 1889.571958] Key type ._llcrypt registered [ 1889.575655] Key type .llcrypt registered [ 1891.625102] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1893.605020] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1895.490315] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 03:40:04 (1779694804) [ 1897.626685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1899.537199] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 03:40:08 (1779694808) [ 1901.527764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg408-server [ 1904.125762] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 03:40:12 (1779694812) [ 1906.695695] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg408-server [ 1909.041898] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 03:40:17 (1779694817) [ 1911.026698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1911.060032] alg: No test for adler32 (adler32-zlib) [ 1912.456901] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 1912.802814] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1914.552979] Key type lgssc registered [ 1916.453893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1940.250496] Key type lgssc unregistered [ 1940.674940] LNet: 96285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1940.687233] LNetError: 96285:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1940.721845] LNet: Removed LNI 192.168.204.8@tcp [ 1941.587482] Key type .llcrypt unregistered [ 1941.590119] Key type ._llcrypt unregistered [ 1956.725959] Key type ._llcrypt registered [ 1956.727860] Key type .llcrypt registered [ 1958.329924] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 03:41:06 (1779694866) [ 1958.784800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1958.803639] alg: No test for adler32 (adler32-zlib) [ 1959.753654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1959.818537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1959.883121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1959.952346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1960.000218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1967.319979] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 03:41:16 (1779694876) [ 1967.396782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1967.422735] LNet: 97692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.463243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1967.577050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1967.606220] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1968.833421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1968.855915] LNet: 97844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1968.860696] LNetError: 97844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1968.875842] LNet: Removed LNI 192.168.204.8@tcp [ 1968.892640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1968.954657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1968.975666] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1969.642604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1969.663326] LNetError: 97872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1969.674949] LNet: Removed LNI 192.168.204.8@tcp [ 1969.736728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1969.824803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1969.845123] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1980.235523] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 03:41:27 (1779694887) [ 1980.639985] LNet: 98463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.646747] LNet: 98463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1980.653981] LNetError: 98463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1980.674238] LNet: Removed LNI 192.168.204.8@tcp [ 1981.455865] Key type .llcrypt unregistered [ 1981.470517] Key type ._llcrypt unregistered [ 1991.960170] Key type ._llcrypt registered [ 1991.964907] Key type .llcrypt registered [ 1992.302969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1992.343757] alg: No test for adler32 (adler32-zlib) [ 1993.284382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.428270] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 1993.435129] LNet: *** cfs_fail_loc=e003, val=100*** [ 1994.785708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 1994.807142] LNet: *** cfs_fail_loc=e003, val=100*** [ 1994.811173] LNet: Skipped 2 previous similar messages [ 1994.826660] Ping reply received with mapping: [ 1994.831914] found metadata reply NI[192.168.204.8@tcp] [ 1994.835696] 08 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1994.839573] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1994.844031] 5a 5a 5a 5a 5a 5a 5a 5a [ 1995.076769] LNet: 98860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1995.100279] LNetError: 98860:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1995.125429] LNet: Removed LNI 192.168.204.8@tcp [ 1996.234629] Key type .llcrypt unregistered [ 1996.241296] Key type ._llcrypt unregistered [ 2009.501463] Key type ._llcrypt registered [ 2009.507503] Key type .llcrypt registered [ 2010.918744] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 03:41:59 (1779694919) [ 2011.558578] Key type .llcrypt unregistered [ 2011.564645] Key type ._llcrypt unregistered [ 2018.041481] Key type ._llcrypt registered [ 2018.046610] Key type .llcrypt registered [ 2018.246608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2018.260235] alg: No test for adler32 (adler32-zlib) [ 2019.354506] LNetError: 99906:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2019.711406] Key type .llcrypt unregistered [ 2019.713468] Key type ._llcrypt unregistered [ 2024.829546] Key type ._llcrypt registered [ 2024.831862] Key type .llcrypt registered [ 2026.482533] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 03:42:14 (1779694934) [ 2027.196407] Key type .llcrypt unregistered [ 2027.198765] Key type ._llcrypt unregistered [ 2035.415096] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2035.421277] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2038.203731] Key type ._llcrypt registered [ 2038.215080] Key type .llcrypt registered [ 2038.496943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2038.524700] alg: No test for adler32 (adler32-zlib) [ 2039.536588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2039.669275] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2052.323261] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 03:42:40 (1779694960) [ 2052.421258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2052.460824] LNet: 101681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.469757] LNetError: 101681:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2052.486795] LNet: Removed LNI 10.1.2.3@tcp [ 2052.528107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2053.404314] Key type .llcrypt unregistered [ 2053.411921] Key type ._llcrypt unregistered [ 2060.247869] Key type ._llcrypt registered [ 2060.253106] Key type .llcrypt registered [ 2060.595472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2060.639149] alg: No test for adler32 (adler32-zlib) [ 2061.686207] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2061.798438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 2061.955810] LNet: 101976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.960146] LNetError: 101976:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2061.971894] LNet: Removed LNI 192.168.204.8@tcp [ 2062.647467] Key type .llcrypt unregistered [ 2062.653443] Key type ._llcrypt unregistered [ 2075.427598] Key type ._llcrypt registered [ 2075.430394] Key type .llcrypt registered [ 2076.944481] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 03:43:05 (1779694985) [ 2077.973403] Key type .llcrypt unregistered [ 2077.977428] Key type ._llcrypt unregistered [ 2086.388401] Key type ._llcrypt registered [ 2086.399681] Key type .llcrypt registered [ 2086.722148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2086.758864] alg: No test for adler32 (adler32-zlib) [ 2095.931498] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 03:43:24 (1779695004) [ 2096.097313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2096.153211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.276659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2096.324397] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2097.438808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2097.484781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2097.535921] LNet: Removed LNI 192.168.204.8@tcp [ 2097.810527] LNet: 103720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2098.148391] Key type .llcrypt unregistered [ 2098.151114] Key type ._llcrypt unregistered [ 2110.791290] Key type ._llcrypt registered [ 2110.799567] Key type .llcrypt registered [ 2112.176687] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 03:43:40 (1779695020) [ 2112.736577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2112.753502] alg: No test for adler32 (adler32-zlib) [ 2113.740445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.821115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2113.854571] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2114.680791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 2114.720684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.8@tcp [ 2114.740472] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2114.746249] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2114.749779] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2114, 0, 0 [ 2114.756729] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2114.759965] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2114.769809] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2114.777199] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2114, 0, 0 [ 2114.783465] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2114.786203] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2114.789857] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2114.792915] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2114, 0, 2164 [ 2114.797708] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2114.803814] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2114.808219] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2114.812337] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2114, 0, 2164 [ 2114.817748] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2121.719905] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 03:43:50 (1779695030) [ 2122.207263] LNet: 105260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2122.212996] LNetError: 105260:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.239254] LNet: Removed LNI 192.168.204.8@tcp [ 2124.412088] Key type .llcrypt unregistered [ 2124.417244] Key type ._llcrypt unregistered [ 2132.170417] Key type ._llcrypt registered [ 2132.172271] Key type .llcrypt registered [ 2132.578389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2132.608139] alg: No test for adler32 (adler32-zlib) [ 2133.574955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2133.641932] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2142.653487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 2154.131968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.226352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2154.280658] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 2155.598304] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2155.807040] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2155.815985] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2155.823628] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2158.260440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2158.301817] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2170.474877] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 03:44:39 (1779695079) [ 2170.605453] LNet: 106755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.620109] LNetError: 106755:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2170.628487] LNet: Removed LNI 192.168.204.8@tcp [ 2171.679240] LNet: Removed LNI 192.168.204.8@tcp1 [ 2171.683935] LNetError: 106755:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2171.697220] LNetError: 106755:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2171.719769] LNet: Removed LNI 10.1.2.3@tcp99 [ 2172.467116] Key type .llcrypt unregistered [ 2172.469492] Key type ._llcrypt unregistered [ 2180.306164] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2180.310163] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2182.908156] Key type ._llcrypt registered [ 2182.910183] Key type .llcrypt registered [ 2183.178326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2183.194217] alg: No test for adler32 (adler32-zlib) [ 2184.160682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2184.225711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2184.255137] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2185.145847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2185.162127] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2188.099587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2188.145074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2188.204690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2188.238142] LNetError: 107508:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2188.282453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2188.323838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2188.795130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2188.950790] LNet: 107529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2188.961074] LNetError: 107529:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2188.971352] LNet: Removed LNI 192.168.204.8@tcp [ 2189.001937] LNet: Removed LNI 10.1.2.3@tcp [ 2189.838225] Key type .llcrypt unregistered [ 2189.841272] Key type ._llcrypt unregistered [ 2203.251299] Key type ._llcrypt registered [ 2203.253553] Key type .llcrypt registered [ 2204.739884] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 03:45:13 (1779695113) [ 2205.549465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2205.570543] alg: No test for adler32 (adler32-zlib) [ 2206.531082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2206.598899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2206.639957] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2207.552808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.8@tcp [ 2207.647704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg408-client@tcp [ 2214.621877] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 03:45:23 (1779695123) [ 2214.739831] LNet: 109057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.777542] LNetError: 109057:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.789920] LNet: Removed LNI 192.168.204.8@tcp [ 2215.374666] Key type .llcrypt unregistered [ 2215.380579] Key type ._llcrypt unregistered [ 2221.198578] Key type ._llcrypt registered [ 2221.201944] Key type .llcrypt registered [ 2221.364838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2221.379820] alg: No test for adler32 (adler32-zlib) [ 2222.317452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2222.344791] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2222.388655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2229.164929] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 03:45:38 (1779695138) [ 2230.651246] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2232.721473] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 03:45:41 (1779695141) [ 2234.416197] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2236.577912] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 03:45:45 (1779695145) [ 2236.708151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2236.738639] LNet: 110477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2236.755320] LNetError: 110477:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2236.796690] LNet: Removed LNI 192.168.204.8@tcp [ 2236.830582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2236.950555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2236.975297] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2237.659662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2237.732743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2245.254485] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 03:45:54 (1779695154) [ 2245.530657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2245.554528] LNet: 111190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2245.562504] LNetError: 111190:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2245.598411] LNet: Removed LNI 192.168.204.8@tcp [ 2245.623147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2245.692734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2245.735321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2246.548645] Key type .llcrypt unregistered [ 2246.551875] Key type ._llcrypt unregistered [ 2259.707461] Key type ._llcrypt registered [ 2259.711456] Key type .llcrypt registered [ 2261.144211] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 03:46:09 (1779695169) [ 2261.665141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2261.692556] alg: No test for adler32 (adler32-zlib) [ 2262.740349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2262.843189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2262.892338] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2263.898727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2263.985561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2271.849300] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 03:46:20 (1779695180) [ 2272.121128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2272.165317] LNet: 112750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.176950] LNetError: 112750:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2272.208252] LNet: Removed LNI 192.168.204.8@tcp [ 2272.245202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.342239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2272.382858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2280.701889] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 03:46:29 (1779695189) [ 2280.916363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2280.946908] LNet: 113330:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.006744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2281.118236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2281.729144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2281.776756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2281.882827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2282.408643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2282.469072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2282.568542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2282.589858] LNet: 113364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2282.617909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.699576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2283.222380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2283.271126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2283.349322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2283.877594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2283.920486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2290.472491] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 03:46:39 (1779695199) [ 2290.603122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2290.656717] LNet: 113957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2290.692030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2290.787163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2290.807615] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2291.740678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2291.778252] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2292.534365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2292.582992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2292.823680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2292.846798] LNetError: 114244:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2293.855256] LNet: Removed LNI 192.168.204.8@tcp [ 2293.861105] LNetError: 114244:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2293.886399] LNet: Removed LNI 192.168.204.8@tcp2 [ 2293.909866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.970358] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2293.974703] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2293.978634] LNet: 114250:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2294.306737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2294.561867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2295.583172] LNet: Removed LNI 192.168.204.8@tcp [ 2295.593806] LNet: Removed LNI 192.168.204.8@tcp2 [ 2295.646805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.759647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2295.827029] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2296.756161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2296.772909] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2297.878829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2297.950166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2298.033296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2298.352731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2298.381783] LNet: 114571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2298.387654] LNet: 114571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2298.394622] LNetError: 114571:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2298.398978] LNetError: 114571:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2298.404736] LNet: Removed LNI 192.168.204.8@tcp [ 2299.432684] LNet: Removed LNI 192.168.204.8@tcp2 [ 2299.483768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.561022] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2299.566609] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2299.571737] LNet: 114577:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2300.004916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2300.205074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2300.228298] LNet: Removed LNI 192.168.204.8@tcp [ 2301.288356] LNet: Removed LNI 192.168.204.8@tcp2 [ 2301.314535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.358972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2301.378953] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2302.026826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2302.046110] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2303.181859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2303.248963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2303.591277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2303.636972] LNetError: 114894:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2303.649535] LNetError: 114894:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2303.664239] LNet: Removed LNI 192.168.204.8@tcp [ 2303.691401] LNet: Removed LNI 192.168.204.8@tcp2 [ 2303.738081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2303.822478] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2303.830752] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2303.841732] LNet: 114900:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2304.351621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2304.597860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2305.631541] LNet: Removed LNI 192.168.204.8@tcp [ 2305.647633] LNet: Removed LNI 192.168.204.8@tcp2 [ 2305.697482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2305.787920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2305.810117] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2306.471586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2306.508170] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2307.401142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2307.437592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2307.494980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2307.786940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2307.831487] LNet: 115222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2307.847139] LNet: 115222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2308.895332] LNet: Removed LNI 192.168.204.8@tcp [ 2309.931563] LNet: Removed LNI 192.168.204.8@tcp2 [ 2309.998955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.099420] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2310.111955] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2310.122451] LNet: 115228:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2310.688603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2310.976310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2311.015423] LNet: Removed LNI 192.168.204.8@tcp [ 2312.042212] LNet: Removed LNI 192.168.204.8@tcp2 [ 2312.083987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.167439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2312.201850] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2313.147365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2313.179540] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2314.048219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2314.092889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2314.171124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2314.590289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2314.618328] LNetError: 115562:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2314.627431] LNetError: 115562:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2314.634896] LNet: Removed LNI 192.168.204.8@tcp [ 2314.666629] LNet: Removed LNI 192.168.204.8@tcp2 [ 2314.729890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2314.806484] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2314.813971] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2314.819373] LNet: 115568:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2315.497629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2315.925370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2316.959180] LNet: Removed LNI 192.168.204.8@tcp [ 2316.969327] LNet: Removed LNI 192.168.204.8@tcp2 [ 2317.041720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2317.193563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2317.231493] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2318.098880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2318.120114] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2319.078203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2319.114940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2319.149813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2319.234672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2319.650158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2320.671728] LNet: Removed LNI 192.168.204.8@tcp [ 2321.712392] LNet: Removed LNI 192.168.204.8@tcp2 [ 2321.763581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2321.840896] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2321.854293] LNet: Added LNI 192.168.204.8@tcp2 [8/256/0/180] [ 2322.460186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2329.799191] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2331.499580] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 03:47:20 (1779695240) [ 2331.784424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2331.804638] LNet: 116686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2331.809468] LNet: 116686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2331.814521] LNetError: 116686:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2331.819746] LNetError: 116686:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2331.823789] LNet: Removed LNI 192.168.204.8@tcp [ 2332.837895] LNet: Removed LNI 192.168.204.8@tcp2 [ 2332.855977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.138817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2333.187190] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2333.933418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2333.954162] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2335.040364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.067981] LNet: Removed LNI 192.168.204.8@tcp [ 2335.083551] LNet: Removed LNI 10.1.2.3@tcp [ 2335.113144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.192601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2335.227268] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2335.232994] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2335.696597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2335.720661] LNet: Removed LNI 192.168.204.8@tcp [ 2335.734269] LNet: Removed LNI 10.1.2.3@tcp [ 2342.519532] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 03:47:31 (1779695251) [ 2342.819646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2342.872795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2342.995989] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2343.075107] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2343.084716] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2343.098454] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2345.572530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2345.607436] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2346.323812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2346.344950] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2348.624844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.8@tcp [ 2348.661933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2348.709366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.8@tcp -m PUT -e local_timeout -r 1 [ 2348.792483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2348.899586] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2348.905862] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2349.173707] LNetError: 117624:0:(lib-move.c:4710:lnet_parse()) 192.168.204.8@tcp, src 192.168.204.8@tcp: Dropping PUT (error -108 looking up sender) [ 2351.199248] LNet: Removed LNI 192.168.204.8@tcp [ 2351.212979] LNet: Removed LNI 10.1.2.3@tcp [ 2351.647347] Key type .llcrypt unregistered [ 2351.651974] Key type ._llcrypt unregistered [ 2364.563842] Key type ._llcrypt registered [ 2364.569629] Key type .llcrypt registered [ 2365.856601] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 03:47:54 (1779695274) [ 2366.347634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2366.376158] alg: No test for adler32 (adler32-zlib) [ 2367.326468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2367.653535] LNet: 118936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2368.255591] Key type .llcrypt unregistered [ 2368.260333] Key type ._llcrypt unregistered [ 2375.229101] Key type ._llcrypt registered [ 2375.233135] Key type .llcrypt registered [ 2375.467708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2375.490294] alg: No test for adler32 (adler32-zlib) [ 2376.451398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2376.491184] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2384.698784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 2384.927379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2385.414723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.108@tcp [ 2385.653691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2385.682328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2385.716852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2385.745688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2385.760164] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.108@tcp [ 2385.778485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2385.807570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2385.843548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2385.885381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2385.913783] LNet: 119386:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2385.932963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2385.971224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2386.013190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2386.070705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2386.107311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2386.156807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2386.838804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2387.080090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2387.106107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2387.134907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2387.165491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2387.216301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2387.256712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2387.300441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2387.332293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2387.359176] LNet: 119503:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2387.389883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2387.429132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2387.467431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2387.513586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2387.563156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2387.604909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2388.223708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.108@tcp [ 2388.572762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2388.623973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2388.673897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2388.732616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2388.779475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2388.809221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2388.857784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2388.889872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2388.907605] LNet: 119620:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2388.928543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2388.969406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2389.004593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2389.061559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2389.103490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2389.154375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2389.882516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2390.137589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2390.175496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2390.199742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2390.224677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2390.274962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2390.324797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2390.359579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2390.416790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2390.474532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2390.523337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2390.569907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2390.618815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2390.672911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2390.716470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2391.072880] LNet: 119784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2391.080550] LNetError: 119784:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2391.098885] LNet: Removed LNI 192.168.204.8@tcp [ 2391.524690] Key type .llcrypt unregistered [ 2391.530083] Key type ._llcrypt unregistered [ 2410.544528] Key type ._llcrypt registered [ 2410.550333] Key type .llcrypt registered [ 2411.920886] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 03:48:40 (1779695320) [ 2412.391902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2412.422863] alg: No test for adler32 (adler32-zlib) [ 2413.346488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2413.716501] LNet: 120671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2414.329077] Key type .llcrypt unregistered [ 2414.334876] Key type ._llcrypt unregistered [ 2421.155635] Key type ._llcrypt registered [ 2421.158434] Key type .llcrypt registered [ 2421.442621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2421.474421] alg: No test for adler32 (adler32-zlib) [ 2422.457483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2422.525986] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2430.328983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.108@tcp [ 2440.552235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.622414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2440.674965] LNet: Added LNI 192.168.204.8@tcp1 [8/256/0/180] [ 2442.115540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2443.105352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.108@tcp [ 2443.532973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2443.575504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2443.650980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2443.724370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2443.774594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2443.834977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2443.876506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2443.913488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2443.933946] LNet: 121323:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2443.970527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2444.009877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2444.059553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2444.069691] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.108@tcp [ 2444.108744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2444.178415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2444.238560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2445.324491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2445.632706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2445.663632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2445.694190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2445.729812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2445.766265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2445.830463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2445.877798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2445.930752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2445.969165] LNet: 121441:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2445.993462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2446.039114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2446.068704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2446.107394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2446.169859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2446.226859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2446.978719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.108@tcp [ 2447.191724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2447.223726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2447.256399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2447.286853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2447.319975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2447.357217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2447.388894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2447.438038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2447.465513] LNet: 121558:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2447.513539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2447.581696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2447.624960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2447.670325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2447.730712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2447.762932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2448.490910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2448.754925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2448.790969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2448.823731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2448.862855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2448.905567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2448.953708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2449.012366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2449.058039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2449.108352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2449.149227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2449.184067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2449.247148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2449.292763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2449.349938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2454.312639] LNet: 121756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2454.317725] LNetError: 121756:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2454.322142] LNet: Removed LNI 192.168.204.8@tcp [ 2454.341130] LNet: Removed LNI 192.168.204.8@tcp1 [ 2454.737879] Key type .llcrypt unregistered [ 2454.743281] Key type ._llcrypt unregistered [ 2471.782887] Key type ._llcrypt registered [ 2471.796525] Key type .llcrypt registered [ 2473.136117] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 03:49:41 (1779695381) [ 2479.716699] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 03:49:48 (1779695388) [ 2487.278134] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 03:49:55 (1779695395) [ 2496.714685] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 03:50:05 (1779695405) [ 2505.574640] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 03:50:14 (1779695414) [ 2513.577804] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 03:50:22 (1779695422) [ 2522.316785] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 03:50:30 (1779695430) [ 2531.999088] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 03:50:40 (1779695440) [ 2540.225155] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 03:50:49 (1779695449) [ 2546.817980] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 03:50:55 (1779695455) [ 2554.513055] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 03:51:03 (1779695463) [ 2562.166779] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 03:51:10 (1779695470) [ 2569.793481] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 03:51:18 (1779695478) [ 2579.818401] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 03:51:27 (1779695487) [ 2591.024221] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 03:51:39 (1779695499) [ 2600.389842] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 03:51:48 (1779695508) [ 2609.870910] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 03:51:58 (1779695518) [ 2618.687381] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 03:52:07 (1779695527) [ 2619.484814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2619.500611] alg: No test for adler32 (adler32-zlib) [ 2620.427622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2620.565709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2620.588421] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2621.799299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2621.813238] LNet: 132470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2621.818323] LNetError: 132470:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2621.829313] LNet: Removed LNI 192.168.204.8@tcp [ 2621.847541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2621.924815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2621.948235] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2628.916563] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 03:52:17 (1779695537) [ 2636.461444] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 03:52:25 (1779695545) [ 2644.060992] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 03:52:32 (1779695552) [ 2652.077927] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 03:52:40 (1779695560) [ 2661.068412] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 03:52:49 (1779695569) [ 2668.933581] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 03:52:57 (1779695577) [ 2676.572247] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 03:53:05 (1779695585) [ 2685.693799] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 03:53:14 (1779695594) [ 2694.499543] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 03:53:22 (1779695602) [ 2703.546531] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 03:53:32 (1779695612) [ 2710.576881] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 03:53:39 (1779695619) [ 2719.015721] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 03:53:47 (1779695627) [ 2725.872881] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2430 sec ========== 03:53:54 (1779695634) [ 2726.164312] LNet: 139937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2726.174744] LNetError: 139937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2726.192483] LNet: Removed LNI 192.168.204.8@tcp [ 2726.729750] Key type .llcrypt unregistered [ 2726.752011] Key type ._llcrypt unregistered [ 2739.626207] Key type ._llcrypt registered [ 2739.639226] Key type .llcrypt registered [ 2740.127561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2740.146414] alg: No test for adler32 (adler32-zlib) [ 2741.249244] Lustre: Lustre: Build Version: 2.17.53_24_gacb3840 [ 2741.476996] LNet: Added LNI 192.168.204.8@tcp [8/256/0/180] [ 2743.175251] Key type lgssc registered [ 2744.530581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2808.667637] Lustre: Mounted lustre-client [ 2813.770410] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2822.256729] LustreError: 142386:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8aea9931c000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2822.319195] Lustre: Unmounted lustre-client [ 2862.540143] Key type lgssc unregistered [ 2862.803738] LNet: 142869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2862.808969] LNetError: 142869:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2862.826410] LNet: Removed LNI 192.168.204.8@tcp [ 2863.400187] Key type .llcrypt unregistered [ 2863.402695] Key type ._llcrypt unregistered