[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 462498554 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002358] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.006952] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008015] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010142] LSM: Security Framework initializing [ 0.012011] Yama: becoming mindful. [ 0.013025] SELinux: Initializing. [ 0.013997] *** VALIDATE selinux *** [ 0.019620] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024756] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025135] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] *** VALIDATE tmpfs *** [ 0.030121] *** VALIDATE proc *** [ 0.031177] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033278] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036027] Spectre V2 : User space: Vulnerable [ 0.037009] Speculative Store Bypass: Vulnerable [ 0.039790] debug: unmapping init [mem 0xffffffffab059000-0xffffffffab060fff] [ 0.041892] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042642] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043021] ... version: 2 [ 0.044009] ... bit width: 48 [ 0.044952] ... generic registers: 4 [ 0.045011] ... value mask: 0000ffffffffffff [ 0.046009] ... max period: 00007fffffffffff [ 0.047009] ... fixed-purpose events: 3 [ 0.048009] ... event mask: 000000070000000f [ 0.049250] rcu: Hierarchical SRCU implementation. [ 0.051395] smp: Bringing up secondary CPUs ... [ 0.052483] x86: Booting SMP configuration: [ 0.053020] .... node #0, CPUs: #1 #2 #3 [ 0.056080] smp: Brought up 1 node, 4 CPUs [ 0.058008] smpboot: Max logical packages: 1 [ 0.059016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.212157] node 0 deferred pages initialised in 150ms [ 0.216133] devtmpfs: initialized [ 0.217345] x86/mm: Memory block size: 128MB [ 0.221142] gcov: version magic: 0x41383552 [ 0.224313] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.228125] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.232410] pinctrl core: initialized pinctrl subsystem [ 0.234210] [ 0.234891] ************************************************************* [ 0.237018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.240020] ** ** [ 0.243013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.245014] ** ** [ 0.248014] ** This means that this kernel is built to expose internal ** [ 0.250010] ** IOMMU data structures, which may compromise security on ** [ 0.252011] ** your system. ** [ 0.254013] ** ** [ 0.256011] ** If you see this message and you are not debugging the ** [ 0.258010] ** kernel, report this immediately to your vendor! ** [ 0.260013] ** ** [ 0.262009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.263006] ************************************************************* [ 0.264552] NET: Registered protocol family 16 [ 0.266288] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.268054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.269047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.272100] cpuidle: using governor menu [ 0.273904] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.276303] PCI: Using configuration type 1 for base access [ 0.278124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.286542] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.287016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.288159] cryptd: max_cpu_qlen set to 1000 [ 0.292293] ACPI: Added _OSI(Module Device) [ 0.293013] ACPI: Added _OSI(Processor Device) [ 0.295014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.299016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.302977] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.309504] ACPI: Interpreter enabled [ 0.310054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.311011] ACPI: Using IOAPIC for interrupt routing [ 0.313110] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.315386] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.324000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.326040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.328015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.331082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.335083] acpiphp: Slot [2] registered [ 0.336134] acpiphp: Slot [5] registered [ 0.338197] acpiphp: Slot [6] registered [ 0.339098] acpiphp: Slot [3] registered [ 0.340107] acpiphp: Slot [4] registered [ 0.342091] acpiphp: Slot [7] registered [ 0.343105] acpiphp: Slot [8] registered [ 0.345104] acpiphp: Slot [9] registered [ 0.346109] acpiphp: Slot [10] registered [ 0.347103] acpiphp: Slot [11] registered [ 0.348075] acpiphp: Slot [12] registered [ 0.350088] acpiphp: Slot [13] registered [ 0.351079] acpiphp: Slot [14] registered [ 0.352090] acpiphp: Slot [15] registered [ 0.353100] acpiphp: Slot [16] registered [ 0.355112] acpiphp: Slot [17] registered [ 0.356085] acpiphp: Slot [18] registered [ 0.357099] acpiphp: Slot [19] registered [ 0.359132] acpiphp: Slot [20] registered [ 0.360000] acpiphp: Slot [21] registered [ 0.360180] acpiphp: Slot [22] registered [ 0.361150] acpiphp: Slot [23] registered [ 0.363152] acpiphp: Slot [24] registered [ 0.365152] acpiphp: Slot [25] registered [ 0.367169] acpiphp: Slot [26] registered [ 0.368468] acpiphp: Slot [27] registered [ 0.370200] acpiphp: Slot [28] registered [ 0.372185] acpiphp: Slot [29] registered [ 0.374133] acpiphp: Slot [30] registered [ 0.375141] acpiphp: Slot [31] registered [ 0.377117] PCI host bridge to bus 0000:00 [ 0.379031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.381046] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.384034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.387030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.390048] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.393048] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.395230] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.399133] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.400000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.407503] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.412064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.416029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.419026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.422028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.424730] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.428195] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.431065] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.435068] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.438947] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.449675] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.453019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.458656] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.463020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.468010] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.486022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.497569] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.505022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.514019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.533020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.547517] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.549575] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.552510] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.555539] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.558269] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.564072] iommu: Default domain type: Passthrough [ 0.566634] SCSI subsystem initialized [ 0.568179] ACPI: bus type USB registered [ 0.569156] usbcore: registered new interface driver usbfs [ 0.572123] usbcore: registered new interface driver hub [ 0.574135] usbcore: registered new device driver usb [ 0.576226] pps_core: LinuxPPS API ver. 1 registered [ 0.578017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.581092] PTP clock support registered [ 0.583173] EDAC MC: Ver: 3.0.0 [ 0.585178] PCI: Using ACPI for IRQ routing [ 0.586788] NetLabel: Initializing [ 0.588023] NetLabel: domain hash size = 128 [ 0.590016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.592129] NetLabel: unlabeled traffic allowed by default [ 0.594348] vgaarb: loaded [ 0.596238] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.597018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.608050] clocksource: Switched to clocksource kvm-clock [ 0.720423] VFS: Disk quotas dquot_6.6.0 [ 0.722384] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.725498] *** VALIDATE ramfs *** [ 0.727035] *** VALIDATE hugetlbfs *** [ 0.730403] pnp: PnP ACPI init [ 0.733407] pnp: PnP ACPI: found 6 devices [ 0.755143] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.760024] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.761605] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.763866] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.766705] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.769945] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.773209] NET: Registered protocol family 2 [ 0.776455] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.780799] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.784259] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.790063] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.793865] TCP: Hash tables configured (established 65536 bind 65536) [ 0.797104] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.800959] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.804400] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.807859] NET: Registered protocol family 1 [ 0.811586] RPC: Registered named UNIX socket transport module. [ 0.814020] RPC: Registered udp transport module. [ 0.816124] RPC: Registered tcp transport module. [ 0.818107] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.820834] NET: Registered protocol family 44 [ 0.822839] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.825132] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.827496] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.830188] PCI: CLS 0 bytes, default 64 [ 0.831980] Unpacking initramfs... [ 2.249991] debug: unmapping init [mem 0xffff9d5e3cc64000-0xffff9d5e3ffcffff] [ 2.254413] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.257027] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.261345] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.785078] Initialise system trusted keyrings [ 2.788113] Key type blacklist registered [ 2.790354] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.800192] zbud: loaded [ 2.804183] *** VALIDATE nfs *** [ 2.805603] *** VALIDATE nfs4 *** [ 2.807981] pstore: using deflate compression [ 2.813061] Platform Keyring initialized [ 2.911552] NET: Registered protocol family 38 [ 2.914053] Key type asymmetric registered [ 2.916428] Asymmetric key parser 'x509' registered [ 2.918367] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.922659] io scheduler mq-deadline registered [ 2.924605] io scheduler kyber registered [ 2.926416] io scheduler bfq registered [ 2.928418] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.931501] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.935246] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.938612] ACPI: Power Button [PWRF] [ 2.944210] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.951672] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.962920] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.990516] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.019220] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.025741] Non-volatile memory driver v1.3 [ 3.028667] Linux agpgart interface v0.103 [ 3.068062] virtio_blk virtio1: [vda] 145936 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.070711] vda: detected capacity change from 0 to 74719232 [ 3.085884] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.089318] vdb: detected capacity change from 0 to 1073741824 [ 3.097447] libphy: Fixed MDIO Bus: probed [ 3.104825] usbcore: registered new interface driver usbserial_generic [ 3.107559] usbserial: USB Serial support registered for generic [ 3.110353] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.115219] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.117590] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.120169] mousedev: PS/2 mouse device common for all mice [ 3.122626] rtc_cmos 00:05: RTC can wake from S4 [ 3.125216] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.125493] rtc_cmos 00:05: registered as rtc0 [ 3.129466] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.132842] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.133807] intel_pstate: CPU model not supported [ 3.138882] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.140588] hid: raw HID events driver (C) Jiri Kosina [ 3.144541] usbcore: registered new interface driver usbhid [ 3.146835] usbhid: USB HID core driver [ 3.148819] drop_monitor: Initializing network drop monitor service [ 3.150529] Initializing XFRM netlink socket [ 3.152209] NET: Registered protocol family 10 [ 3.154728] Segment Routing with IPv6 [ 3.155802] NET: Registered protocol family 17 [ 3.158232] mpls_gso: MPLS GSO support [ 3.164313] RAS: Correctable Errors collector initialized. [ 3.167797] AVX version of gcm_enc/dec engaged. [ 3.168869] AES CTR mode by8 optimization enabled [ 3.239935] sched_clock: Marking stable (3239909618, 0)->(4106377244, -866467626) [ 3.244908] registered taskstats version 1 [ 3.247938] Loading compiled-in X.509 certificates [ 3.250721] zswap: loaded using pool lzo/zbud [ 3.275832] Key type big_key registered [ 3.289735] Key type encrypted registered [ 3.291611] ima: No TPM chip found, activating TPM-bypass! [ 3.294086] ima: Allocated hash algorithm: sha1 [ 3.295841] ima: No architecture policies found [ 3.297793] evm: Initialising EVM extended attributes: [ 3.300129] evm: security.selinux [ 3.301377] evm: security.ima [ 3.302744] evm: security.capability [ 3.304274] evm: HMAC attrs: 0x1 [ 3.307956] rtc_cmos 00:05: setting system clock to 2026-07-24 23:31:27 UTC (1784935887) [ 3.314242] debug: unmapping init [mem 0xffffffffac003000-0xffffffffac1fffff] [ 3.317374] debug: unmapping init [mem 0xffffffffaad82000-0xffffffffab058fff] [ 3.326153] Write protecting the kernel read-only data: 28672k [ 3.329021] debug: unmapping init [mem 0xffffffffa9403000-0xffffffffa95fffff] [ 3.331636] debug: unmapping init [mem 0xffffffffa9d14000-0xffffffffa9dfffff] [ 3.363805] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.374186] systemd[1]: Detected virtualization kvm. [ 3.376570] systemd[1]: Detected architecture x86-64. [ 3.378781] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.401578] systemd[1]: No hostname configured. [ 3.402871] systemd[1]: Set hostname to . [ 3.404260] random: systemd: uninitialized urandom read (16 bytes read) [ 3.407079] systemd[1]: Initializing machine ID from random generator. [ 3.540724] random: systemd: uninitialized urandom read (16 bytes read) [ 3.543947] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.549146] random: systemd: uninitialized urandom read (16 bytes read) [ 3.551586] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.555775] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.261370] device-mapper: uevent: version 1.0.3 [ 4.263889] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.035212] virtio_net virtio0 ens2: renamed from eth0 [ 5.101889] scsi host0: ata_piix [ 5.131204] scsi host1: ata_piix [ 5.136320] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.139099] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.403651] dracut-initqueue[591]: RTNETLINK answers: File exists [ 9.810699] random: crng init done [ 9.812382] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.244171] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.410266] printk: systemd: 20 output lines suppressed due to ratelimiting [ 11.679981] SELinux: Disabled at runtime. [ 11.734354] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.742933] systemd[1]: Detected virtualization kvm. [ 11.744981] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.331826] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.335956] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.344206] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.350154] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.353135] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.362275] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.368259] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Stopped target S[ 12.434326] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS witch Root. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.891290] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.225465] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.320920] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.416384] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.432598] EDAC sbridge: Ver: 1.1.2 [ 14.683553] Key type dns_resolver registered [ 14.990614] NFS: Registering the id_resolver key type [ 14.992741] Key type id_resolver registered [ 14.994269] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg630-client login: [ 66.399185] hrtimer: interrupt took 6191436 ns [ 73.703947] libcfs: loading out-of-tree module taints kernel. [ 73.936851] Key type ._llcrypt registered [ 73.941822] Key type .llcrypt registered [ 74.475802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 74.507439] alg: No test for adler32 (adler32-zlib) [ 75.995908] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 77.095574] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 79.025363] Key type lgssc registered [ 80.876991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 204.722099] Lustre: Mounted lustre-client [ 209.572225] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 221.267885] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing check_logdir /tmp/testlogs/ [ 226.570365] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing yml_node [ 230.368949] Lustre: lustre-OST0000-osc-ffff9d5e850d3800: disconnect after 23s idle [ 230.740481] Lustre: DEBUG MARKER: Client: 2.17.54.207 [ 233.213213] Lustre: DEBUG MARKER: MDS: 2.17.54.207 [ 235.722978] Lustre: DEBUG MARKER: OSS: 2.17.54.207 [ 237.210878] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Jul 24 19:35:20 EDT 2026 [ 252.280345] Lustre: Unmounted lustre-client [ 282.034479] Key type lgssc unregistered [ 282.386931] LNet: 8389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 282.409509] LNetError: 8389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 282.438848] LNet: Removed LNI 192.168.206.30@tcp [ 283.166444] Key type .llcrypt unregistered [ 283.175047] Key type ._llcrypt unregistered [ 313.952846] Key type ._llcrypt registered [ 313.955569] Key type .llcrypt registered [ 314.210726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 314.230067] alg: No test for adler32 (adler32-zlib) [ 315.209794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 315.268213] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 315.321499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 319.092701] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 319.278772] LNet: 9272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.290837] LNetError: 9272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 319.306955] LNet: Removed LNI 192.168.206.30@tcp [ 320.028568] Key type .llcrypt unregistered [ 320.030934] Key type ._llcrypt unregistered [ 326.531552] Key type ._llcrypt registered [ 326.534947] Key type .llcrypt registered [ 327.974713] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 19:36:50 (1784936210) [ 328.342345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 328.370202] alg: No test for adler32 (adler32-zlib) [ 329.346827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.489722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.512264] LNet: 9685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.530905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.582251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 329.605317] LNet: 9692:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 337.038491] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 19:36:59 (1784936219) [ 337.267732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.301077] LNet: 10269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.362424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.464059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 344.284442] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 19:37:07 (1784936227) [ 344.396817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.448392] LNet: 10852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.493444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.622926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 350.990221] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 19:37:13 (1784936233) [ 351.176315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.196658] LNet: 11442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.212759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.303853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 357.316564] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 19:37:20 (1784936240) [ 357.471345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.493240] LNet: 12026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.514159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.570541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 358.096375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 365.041598] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 19:37:27 (1784936247) [ 365.292310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.395849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.495445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 366.000178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 366.575876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 373.858250] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 19:37:36 (1784936256) [ 373.938255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.984893] LNet: 13205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.994082] LNet: 13205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 374.041213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.186995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 380.645679] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 19:37:43 (1784936263) [ 380.910281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.975930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.045826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 381.077777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 381.529782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 381.589895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 382.150269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 382.189789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 382.860647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 382.920609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 383.442928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 383.499723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 384.063400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 384.110549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 384.581467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 384.636314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 385.153647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 385.197536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 385.663285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 385.696506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 386.255957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 386.302447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 386.829363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 386.888686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 387.407450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 387.454230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 387.938704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 387.973594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 394.211207] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 19:37:57 (1784936277) [ 394.316942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.352169] LNet: 14475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.359983] LNet: 14475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 394.388541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.452974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 394.481937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 400.559225] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 19:38:03 (1784936283) [ 400.790582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.841656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.933247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 400.971907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 407.345563] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 19:38:10 (1784936290) [ 407.459741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.513963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.624715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 407.666380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 413.369464] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 19:38:16 (1784936296) [ 413.567347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.599561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.675564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 413.718038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 419.745682] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 19:38:22 (1784936302) [ 419.896917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.947247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.023281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 420.061594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 425.585465] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 19:38:28 (1784936308) [ 425.920407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.972460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.042295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 426.076477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 432.350427] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 19:38:34 (1784936314) [ 432.416553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.459257] LNet: 17987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.472220] LNet: 17987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 432.517288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.675819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 432.997301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 439.645770] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 19:38:42 (1784936322) [ 439.899391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.953265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.210700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 440.437939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 446.322563] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 19:38:49 (1784936329) [ 446.573339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.634944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.763324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 446.959816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 453.984925] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 19:38:56 (1784936336) [ 454.067138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.101714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.492175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 454.788095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 461.519987] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 19:39:04 (1784936344) [ 461.798891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.833838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.887283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.920838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 468.733779] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 19:39:11 (1784936351) [ 468.887442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.945512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.091815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 469.200877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 475.721039] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 19:39:18 (1784936358) [ 475.976795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.010251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.080943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 481.950454] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 19:39:24 (1784936364) [ 482.052920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.090598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.149724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 488.710243] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 19:39:31 (1784936371) [ 488.818872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.864625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.942212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 489.208237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 496.377035] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 19:39:39 (1784936379) [ 496.479731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.528285] LNet: 23430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 496.542430] LNet: 23430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 496.582923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.732879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 496.767852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 503.730964] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 19:39:46 (1784936386) [ 503.836966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.909222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.015396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 504.061632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 510.191531] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 19:39:53 (1784936393) [ 510.351993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.402939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.489968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 510.537881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 516.464934] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 19:39:59 (1784936399) [ 516.604068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.632199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.719841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 516.766354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 524.702889] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 19:40:06 (1784936406) [ 524.810548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.879668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.973308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 525.074157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 525.099389] LNetError: 25794:0:(peer.c:2277:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 525.164254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 525.204275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 533.314899] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 19:40:15 (1784936415) [ 533.493145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.582744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.699419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XE2taFh6Xu [ 533.807723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XE2taFh6Xu [ 533.872483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XE2taFh6Xu [ 541.131395] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 19:40:23 (1784936423) [ 541.216320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.274994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.375473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 541.428846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 541.495607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 549.107254] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 19:40:31 (1784936431) [ 549.357353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.455989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.590070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 549.616104] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 551.407824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 552.182848] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 554.090454] Key type lgssc registered [ 556.134894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 561.685774] LNetError: 27557:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.30@tcp99 has no available nets [ 564.256790] Key type lgssc unregistered [ 564.633823] LNetError: 28666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 564.653504] LNet: Removed LNI 192.168.206.30@tcp [ 565.224807] Key type .llcrypt unregistered [ 565.227756] Key type ._llcrypt unregistered [ 572.383289] Key type ._llcrypt registered [ 572.385795] Key type .llcrypt registered [ 573.534783] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 19:40:56 (1784936456) [ 574.104461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 574.126921] alg: No test for adler32 (adler32-zlib) [ 575.058323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.160831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 577.693918] LNet: Message forwarding will be enabled in 2s [ 580.768121] LNet: Message forwarding enabled [ 581.146854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 581.189083] LNet: Message forwarding will stop in 2s to 6s [ 584.503446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 585.137054] LNet: Message forwarding disabled [ 586.037896] LNet: Message forwarding will be enabled in 2s [ 589.221108] LNet: Message forwarding enabled [ 595.324545] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 19:41:17 (1784936477) [ 595.475956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.537683] LNet: 30061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.356409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.478947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 605.908623] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 19:41:28 (1784936488) [ 606.067881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.102947] LNet: 30647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.141972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.198970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 606.227876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 606.263795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 606.304566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 606.333390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 606.356901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 606.392400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 606.443717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 606.490331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 606.550851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 606.580633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 606.603889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 606.647118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 606.699653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 606.732934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 606.793672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 606.843840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 606.882730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 606.921507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 606.947012] LNetError: 30690:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 614.350956] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 19:41:36 (1784936496) [ 614.490207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.545869] LNet: 31255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.599863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.754371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 623.448865] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 19:41:46 (1784936506) [ 625.135370] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg630-server [ 627.144850] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 19:41:49 (1784936509) [ 627.266471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.293099] LNet: 32188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.339146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.442130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 627.505671] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 628.414683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[31-35]@tcp [ 628.461730] LNet: 32325:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 628.525369] LNetError: 32196:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.32@tcp is being used as a gateway but routing feature is not turned on [ 628.547296] LNetError: 32196:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.32@tcp has gone from up to down [ 629.540109] LNetError: 32196:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.32@tcp is being used as a gateway but routing feature is not turned on [ 629.552855] LNetError: 32196:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 630.566628] LNetError: 32196:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.34@tcp is being used as a gateway but routing feature is not turned on [ 630.577544] LNetError: 32196:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 632.616808] LNetError: 32196:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.31@tcp is being used as a gateway but routing feature is not turned on [ 632.632893] LNetError: 32196:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 635.206386] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 19:41:57 (1784936517) [ 636.718214] LNetError: 32196:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.35@tcp is being used as a gateway but routing feature is not turned on [ 636.736200] LNetError: 32196:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 637.213214] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg630-server [ 639.151674] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 19:42:02 (1784936522) [ 639.247911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.276874] LNet: 33258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.286455] LNetError: 33258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 639.326303] LNet: Removed LNI 192.168.206.30@tcp [ 639.367945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.429726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 639.460822] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 640.170433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[31-36/2]@tcp [ 640.193491] LNet: 33379:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 640.203317] LNet: 33379:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 640.234221] LNetError: 33266:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.31@tcp has gone from up to down [ 640.244729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[31-36/2]@tcp [ 640.253945] LNetError: 33266:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 646.430756] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 19:42:09 (1784936529) [ 646.790881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.842521] LNetError: 33972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 646.868796] LNet: Removed LNI 192.168.206.30@tcp [ 646.902484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.048139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 647.110373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.148954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.205435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 647.264455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.284364] LNet: 33998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.291342] LNet: 33998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 647.312764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.364262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 647.429953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.459500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.520299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 647.598774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.626659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.680966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 647.756405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.787887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.832566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 647.899744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.926897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.001155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 648.106642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.153691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.266934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 648.356908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.397564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.477136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 648.572826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.606678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.701250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 648.799750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.832692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.878107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 648.950552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.983104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.050342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 654.851839] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 19:42:17 (1784936537) [ 656.412326] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg630-server [ 658.043609] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 19:42:20 (1784936540) [ 659.955634] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg630-server [ 661.748263] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 19:42:24 (1784936544) [ 662.619806] Key type .llcrypt unregistered [ 662.622674] Key type ._llcrypt unregistered [ 669.002612] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 669.221260] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 669.233136] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 669.240200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 671.810965] Key type ._llcrypt registered [ 671.815176] Key type .llcrypt registered [ 672.028804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 672.055919] alg: No test for adler32 (adler32-zlib) [ 673.095176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.188613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 673.227523] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 673.912100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 673.939509] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 676.220794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 676.251431] LNet: Removed LNI 10.1.2.3@tcp [ 676.392278] LNet: 36096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.405050] LNetError: 36096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.424888] LNet: Removed LNI 192.168.206.30@tcp [ 677.024386] Key type .llcrypt unregistered [ 677.033317] Key type ._llcrypt unregistered [ 689.584684] Key type ._llcrypt registered [ 689.587289] Key type .llcrypt registered [ 690.761989] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 19:42:53 (1784936573) [ 691.299955] Key type .llcrypt unregistered [ 691.311967] Key type ._llcrypt unregistered [ 697.645588] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 697.856449] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 697.864321] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 697.871373] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 700.557820] Key type ._llcrypt registered [ 700.560593] Key type .llcrypt registered [ 700.780505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 700.804432] alg: No test for adler32 (adler32-zlib) [ 701.723274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.786435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 701.824658] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 702.558654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 702.583055] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 705.147828] LNet: 37591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.152158] LNetError: 37591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 705.162742] LNet: Removed LNI 10.1.2.3@tcp [ 705.173269] LNet: Removed LNI 192.168.206.30@tcp [ 705.810545] Key type .llcrypt unregistered [ 705.818123] Key type ._llcrypt unregistered [ 717.730468] Key type ._llcrypt registered [ 717.732105] Key type .llcrypt registered [ 719.448237] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 19:43:21 (1784936601) [ 719.957936] Key type .llcrypt unregistered [ 719.959751] Key type ._llcrypt unregistered [ 725.436246] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 725.535162] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 725.545305] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 725.549312] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 728.267981] Key type ._llcrypt registered [ 728.270310] Key type .llcrypt registered [ 728.566618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 728.586097] alg: No test for adler32 (adler32-zlib) [ 729.560869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.612609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 729.649946] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 731.522347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 731.545438] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 733.573830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 733.607811] LNet: Removed LNI 10.1.2.3@tcp [ 733.629641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 733.668399] LNet: Removed LNI 10.1.2.31@tcp [ 733.707777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 733.744867] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 734.373151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 734.391937] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 735.378887] LNet: 39514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.383984] LNetError: 39514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 735.389351] LNet: Removed LNI 10.1.2.31@tcp [ 735.398200] LNet: Removed LNI 10.1.2.3@tcp [ 736.091679] Key type .llcrypt unregistered [ 736.102818] Key type ._llcrypt unregistered [ 747.254479] Key type ._llcrypt registered [ 747.256837] Key type .llcrypt registered [ 748.544646] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 19:43:51 (1784936631) [ 748.833238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 748.848271] alg: No test for adler32 (adler32-zlib) [ 749.724615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.775709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 749.794347] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 750.538672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.570718] LNet: 40468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.578604] LNetError: 40468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 750.597106] LNet: Removed LNI 192.168.206.30@tcp [ 750.613559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.668802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.30@tcp [ 750.702101] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 751.378129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.30@tcp [ 751.418860] LNet: Removed LNI 192.168.206.30@tcp [ 751.510722] LNet: 40603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.170502] Key type .llcrypt unregistered [ 752.174733] Key type ._llcrypt unregistered [ 762.219820] Key type ._llcrypt registered [ 762.222454] Key type .llcrypt registered [ 763.971803] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 19:44:06 (1784936646) [ 766.455801] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg630-server [ 768.246275] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 19:44:10 (1784936650) [ 768.834464] Key type .llcrypt unregistered [ 768.839287] Key type ._llcrypt unregistered [ 775.326645] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 775.552709] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 775.556066] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 775.559741] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 778.392337] Key type ._llcrypt registered [ 778.403699] Key type .llcrypt registered [ 778.714832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 778.747708] alg: No test for adler32 (adler32-zlib) [ 779.638345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.688946] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 780.457161] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 782.737581] LNet: 42441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.743056] LNetError: 42441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.749402] LNet: Removed LNI 192.168.206.30@tcp1 [ 782.757661] LNet: Removed LNI 10.1.2.3@tcp2 [ 783.403447] Key type .llcrypt unregistered [ 783.406241] Key type ._llcrypt unregistered [ 795.250589] Key type ._llcrypt registered [ 795.253284] Key type .llcrypt registered [ 796.624801] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 19:44:39 (1784936679) [ 797.061943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 797.085098] alg: No test for adler32 (adler32-zlib) [ 798.186149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.298248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 798.366630] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 799.411591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.439792] LNet: 43412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.452358] LNetError: 43412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.482307] LNet: Removed LNI 192.168.206.30@tcp [ 799.519213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.659577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 799.708971] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 807.190049] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 19:44:49 (1784936689) [ 807.298759] LNet: 43997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.314368] LNetError: 43997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 807.344854] LNet: Removed LNI 192.168.206.30@tcp [ 807.948800] Key type .llcrypt unregistered [ 807.954044] Key type ._llcrypt unregistered [ 813.208398] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 813.420785] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 813.431818] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 813.440821] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 816.028925] Key type ._llcrypt registered [ 816.032282] Key type .llcrypt registered [ 816.273418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 816.289229] alg: No test for adler32 (adler32-zlib) [ 817.231464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.291261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 817.322328] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 818.313377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 818.343520] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 820.835973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.867393] LNet: 44742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.874296] LNetError: 44742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 820.882281] LNet: Removed LNI 10.1.2.3@tcp [ 820.897197] LNet: Removed LNI 192.168.206.30@tcp [ 820.923348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.004294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 821.041535] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 821.049994] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 821.540687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.559067] LNet: 44769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.566996] LNetError: 44769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 821.576781] LNetError: 44769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 821.585857] LNet: Removed LNI 10.1.2.3@tcp [ 821.602198] LNet: Removed LNI 192.168.206.30@tcp [ 821.624613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.689803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 821.716096] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 821.721044] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 822.040718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 822.061938] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 823.140444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 823.164991] LNet: 44931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.174705] LNetError: 44931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 823.181310] LNetError: 44931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 823.187890] LNet: Removed LNI 10.1.2.3@tcp [ 823.191558] LNet: Removed LNI 192.168.206.30@tcp [ 823.211695] LNet: Removed LNI 192.168.206.30@tcp2 [ 823.245766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.322334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 823.372650] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 823.378982] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 823.384809] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 831.249112] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 19:45:14 (1784936714) [ 831.374902] LNet: 45524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.384937] LNetError: 45524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 831.391932] LNetError: 45524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 831.401927] LNet: Removed LNI 10.1.2.3@tcp [ 831.410575] LNet: Removed LNI 192.168.206.30@tcp [ 831.430426] LNet: Removed LNI 192.168.206.30@tcp2 [ 832.222837] Key type .llcrypt unregistered [ 832.226887] Key type ._llcrypt unregistered [ 838.443794] Key type ._llcrypt registered [ 838.446735] Key type .llcrypt registered [ 838.717448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 838.736393] alg: No test for adler32 (adler32-zlib) [ 839.688494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.785967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 839.837303] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 840.674607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.695204] LNet: 45924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.708590] LNetError: 45924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.725702] LNet: Removed LNI 192.168.206.30@tcp [ 840.780752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.845842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 840.890469] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 841.499452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.520451] LNet: 45954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.525911] LNetError: 45954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.568756] LNet: Removed LNI 192.168.206.30@tcp [ 841.590733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.706975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 841.732704] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 842.317391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.358162] LNet: Removed LNI 192.168.206.30@tcp [ 842.402616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.470503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 842.500345] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 843.072140] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 843.288214] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 843.292135] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 843.298953] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 845.658102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.674668] LNet: 46062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.678104] LNet: 46062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 845.695221] LNetError: 46062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.700210] LNetError: 46062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 845.721987] LNet: Removed LNI 192.168.206.30@tcp [ 845.750198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.825473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 845.848203] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 852.211884] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 19:45:35 (1784936735) [ 852.423623] LNet: 46659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.435121] LNetError: 46659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.458754] LNet: Removed LNI 192.168.206.30@tcp [ 853.123385] Key type .llcrypt unregistered [ 853.128346] Key type ._llcrypt unregistered [ 859.192495] Key type ._llcrypt registered [ 859.194760] Key type .llcrypt registered [ 859.420320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 859.450168] alg: No test for adler32 (adler32-zlib) [ 860.404795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.465355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 860.490927] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 861.465531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.500094] LNet: 47058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.517347] LNetError: 47058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.533691] LNet: Removed LNI 192.168.206.30@tcp [ 861.565394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.605612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 861.630098] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 867.811749] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 19:45:50 (1784936750) [ 867.909638] LNet: 47642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.914763] LNetError: 47642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.932350] LNet: Removed LNI 192.168.206.30@tcp [ 868.552753] Key type .llcrypt unregistered [ 868.555609] Key type ._llcrypt unregistered [ 873.773073] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 873.837949] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 873.844142] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 873.849893] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 876.569906] Key type ._llcrypt registered [ 876.571905] Key type .llcrypt registered [ 876.779128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 876.812814] alg: No test for adler32 (adler32-zlib) [ 877.840292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.930516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 877.959442] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 879.127842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.148992] LNet: 48102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.156962] LNetError: 48102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.173748] LNet: Removed LNI 192.168.206.30@tcp [ 879.205510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.291470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 879.359699] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 886.071330] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 19:46:08 (1784936768) [ 886.223306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 886.256308] LNet: 48699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.266295] LNetError: 48699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.299350] LNet: Removed LNI 192.168.206.30@tcp [ 886.335367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.475546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 893.221601] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 19:46:15 (1784936775) [ 893.449800] LNet: 49282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.274489] Key type .llcrypt unregistered [ 894.282244] Key type ._llcrypt unregistered [ 900.903973] Key type ._llcrypt registered [ 900.906677] Key type .llcrypt registered [ 901.143890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 901.155689] alg: No test for adler32 (adler32-zlib) [ 902.132657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.197257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 902.239927] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 903.367914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 903.393751] LNet: 49680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.404884] LNetError: 49680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.417124] LNet: Removed LNI 192.168.206.30@tcp [ 903.436168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.496477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 903.537721] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 910.785444] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 19:46:33 (1784936793) [ 910.971356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.022316] LNet: 50271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.028450] LNetError: 50271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.079257] LNet: Removed LNI 192.168.206.30@tcp [ 911.116844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.201520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 917.654775] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 19:46:40 (1784936800) [ 919.171584] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 920.965129] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 19:46:43 (1784936803) [ 922.641033] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 924.071042] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 19:46:47 (1784936807) [ 925.942864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 928.190667] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 19:46:50 (1784936810) [ 930.103803] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 932.711247] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 19:46:54 (1784936814) [ 934.803121] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 937.111171] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 19:46:59 (1784936819) [ 937.243806] LNet: 52319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.226476] Key type .llcrypt unregistered [ 938.228711] Key type ._llcrypt unregistered [ 944.930422] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 945.128838] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 945.133086] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 945.146466] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 947.842773] Key type ._llcrypt registered [ 947.848087] Key type .llcrypt registered [ 948.059243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 948.077751] alg: No test for adler32 (adler32-zlib) [ 948.993337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.127717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 949.153784] LNet: Added LNI 192.168.206.30@tcp [8/512/0/180] [ 949.164650] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 949.168872] LNet: 52649:0:(api-ni.c:2987:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 949.252222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.269775] LNet: 52668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.278879] LNetError: 52668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 949.286066] LNet: Removed LNI 10.1.2.3@tcp [ 949.288869] LNetError: 52668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.305556] LNet: Removed LNI 192.168.206.30@tcp [ 949.329337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.366795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 949.386585] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 949.437210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.453694] LNet: 52692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.459607] LNetError: 52692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.474886] LNet: Removed LNI 192.168.206.30@tcp [ 949.493370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.556461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 949.577085] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 949.582767] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 949.585852] LNet: 52700:0:(api-ni.c:2987:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 949.663223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.683794] LNet: 52718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.687912] LNetError: 52718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 949.691607] LNet: Removed LNI 10.1.2.3@tcp [ 949.695568] LNetError: 52718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.713913] LNet: Removed LNI 192.168.206.30@tcp [ 949.730415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.784746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 949.815730] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 949.826486] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 949.828436] LNet: 52726:0:(api-ni.c:2987:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 950.002219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.021479] LNet: 52752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.025757] LNetError: 52752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 950.030300] LNet: Removed LNI 10.1.2.3@tcp [ 950.031913] LNetError: 52752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.045486] LNet: Removed LNI 192.168.206.30@tcp [ 950.064703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.127682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 950.151325] LNet: Added LNI 192.168.206.30@tcp [8/256/0/182] [ 950.161539] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 950.321646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.367644] LNet: 52780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.375266] LNetError: 52780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 950.380468] LNet: Removed LNI 10.1.2.3@tcp [ 950.382268] LNetError: 52780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.399125] LNet: Removed LNI 192.168.206.30@tcp [ 950.481575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.610110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 950.678478] LNet: Added LNI 192.168.206.30@tcp [8/256/0/182] [ 950.694170] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 950.701059] LNet: 52788:0:(api-ni.c:2987:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 950.824560] LNet: Removed LNI 10.1.2.3@tcp [ 950.835969] LNet: Removed LNI 192.168.206.30@tcp [ 951.631807] Key type .llcrypt unregistered [ 951.634098] Key type ._llcrypt unregistered [ 965.225205] Key type ._llcrypt registered [ 965.227943] Key type .llcrypt registered [ 966.922487] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 19:47:29 (1784936849) [ 967.410354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 967.428908] alg: No test for adler32 (adler32-zlib) [ 968.302849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.336713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 968.353427] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 969.225874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.30 [ 969.250316] LNet: 53765:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 969.265400] LNet: Removed LNI 192.168.206.30@tcp [ 976.535554] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 19:47:39 (1784936859) [ 976.803718] LNet: 54335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.811817] LNetError: 54335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 976.831905] LNet: Removed LNI 192.168.206.30@tcp [ 977.503821] Key type .llcrypt unregistered [ 977.507652] Key type ._llcrypt unregistered [ 982.570630] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 982.781193] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 985.494296] Key type ._llcrypt registered [ 985.496772] Key type .llcrypt registered [ 985.684066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 985.703642] alg: No test for adler32 (adler32-zlib) [ 987.025389] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 992.746296] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 19:47:55 (1784936875) [ 992.823255] LNet: 55240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.842922] LNetError: 55240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 992.860980] LNet: Removed LNI 10.1.2.3@tcp [ 993.493459] Key type .llcrypt unregistered [ 993.496667] Key type ._llcrypt unregistered [ 999.298287] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 999.552230] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 999.558343] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 999.567393] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1001.182666] Key type ._llcrypt registered [ 1001.185254] Key type .llcrypt registered [ 1001.385345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1001.434713] alg: No test for adler32 (adler32-zlib) [ 1002.418340] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1008.333520] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 19:48:11 (1784936891) [ 1009.706335] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1011.557299] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 19:48:14 (1784936894) [ 1011.763334] LNet: 56424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.789694] LNetError: 56424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1011.831925] LNet: Removed LNI 10.1.2.3@tcp [ 1012.659422] Key type .llcrypt unregistered [ 1012.664622] Key type ._llcrypt unregistered [ 1018.745378] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1019.005760] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1019.022913] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1019.035267] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1021.876681] Key type ._llcrypt registered [ 1021.880709] Key type .llcrypt registered [ 1022.126982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1022.147206] alg: No test for adler32 (adler32-zlib) [ 1023.120792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.197542] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1032.416975] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 19:48:35 (1784936915) [ 1033.040501] LNet: 57657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1033.045759] LNetError: 57657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1033.069612] LNet: Removed LNI 10.1.2.3@tcp [ 1034.202568] Key type .llcrypt unregistered [ 1034.209850] Key type ._llcrypt unregistered [ 1041.522217] Key type ._llcrypt registered [ 1041.524746] Key type .llcrypt registered [ 1041.898255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1041.932749] alg: No test for adler32 (adler32-zlib) [ 1043.016845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1043.084146] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1050.563854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1051.252305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_interrupt -r 1 [ 1051.290656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_interrupt -r 1 [ 1051.331644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1051.361555] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1051.629576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1051.664336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1051.695545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1051.725769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1051.767475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1051.808593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1051.853097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1051.887522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1051.902112] LNet: 58096:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1051.917694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1051.967565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1052.001706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1052.033989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1052.082465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1052.117781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1052.154885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1052.683893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_dropped -r 1 [ 1052.722146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_dropped -r 1 [ 1052.760894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1052.781983] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1052.976247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1053.018454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1053.046051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1053.069028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1053.109317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1053.143496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1053.191688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1053.218823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1053.238973] LNet: 58205:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1053.256618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1053.289356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1053.326149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1053.352345] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1053.364179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1053.396500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1053.429602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1053.456566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1053.968743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_aborted -r 1 [ 1054.017097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_aborted -r 1 [ 1054.070948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1054.092318] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1054.098861] LNet: Skipped 1 previous similar message [ 1054.318948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1054.370046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1054.420680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1054.460059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1054.517229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1054.569749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1054.622522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1054.663482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1054.694535] LNet: 58314:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1054.726328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1054.762583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1054.799470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1054.848510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1054.895133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1054.929994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1054.964696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1055.426839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_no_route -r 1 [ 1055.455674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_no_route -r 1 [ 1055.489629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1055.685899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1055.726478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1055.781747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1055.833940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1055.882676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1055.926376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1055.962806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1056.003336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1056.039329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1056.066667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1056.137243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1056.199133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1056.231799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1056.263749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1056.288784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1056.862667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_timeout -r 1 [ 1056.909888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_timeout -r 1 [ 1056.943072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1056.969916] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1056.978326] LNet: Skipped 1 previous similar message [ 1057.216070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1057.247395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1057.273689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1057.310882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1057.352909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1057.395993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1057.429938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1057.468878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1057.495788] LNet: 58533:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1057.503090] LNet: 58533:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1057.529984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1057.566285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1057.599075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1057.627374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1057.654699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1057.684702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1057.720577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1058.183254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_error -r 1 [ 1058.217772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_error -r 1 [ 1058.259914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1058.468715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1058.491198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1058.512360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1058.550609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1058.592724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1058.624696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1058.653585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1058.675591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1058.709719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1058.752602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1058.781518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1058.827637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1058.872489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1058.902668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1058.929699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1059.140976] LNet: 58683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.148990] LNetError: 58683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.158287] LNet: Removed LNI 192.168.206.30@tcp [ 1059.484650] Key type .llcrypt unregistered [ 1059.487511] Key type ._llcrypt unregistered [ 1076.120943] Key type ._llcrypt registered [ 1076.123213] Key type .llcrypt registered [ 1077.471752] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 19:49:20 (1784936960) [ 1078.371372] Key type .llcrypt unregistered [ 1078.377469] Key type ._llcrypt unregistered [ 1083.901603] Key type ._llcrypt registered [ 1083.906131] Key type .llcrypt registered [ 1084.102620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1084.117759] alg: No test for adler32 (adler32-zlib) [ 1084.980509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1085.001759] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1091.443579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1101.032838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1101.063610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1101.097770] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1102.466960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_interrupt -r 1 [ 1102.507384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_interrupt -r 1 [ 1102.559947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_interrupt -r 1 [ 1102.608702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_interrupt -r 1 [ 1102.639841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_interrupt -r 1 [ 1102.667745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_interrupt -r 1 [ 1102.708794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_interrupt -r 1 [ 1102.746745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_interrupt -r 1 [ 1102.781343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1102.803647] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1102.812604] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1103.013609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1103.075660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1103.104906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1103.126967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1103.151642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1103.180914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1103.219932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1103.246447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1103.275937] LNet: 60152:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1103.302768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1103.332283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1103.367354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1103.388515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1103.416929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1103.445183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1103.471503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1103.893770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_dropped -r 1 [ 1103.927505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_dropped -r 1 [ 1103.957217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_dropped -r 1 [ 1103.984308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_dropped -r 1 [ 1104.008995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_dropped -r 1 [ 1104.039255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_dropped -r 1 [ 1104.069869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_dropped -r 1 [ 1104.097318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_dropped -r 1 [ 1104.121748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1104.136731] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1104.140925] LNet: Skipped 2 previous similar messages [ 1104.317555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.338859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1104.362585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.386323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.429470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.468868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.503854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.545374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.565267] LNet: 60273:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.585709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.619159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.656975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.720914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.767230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.822742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.871374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1105.388650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_aborted -r 1 [ 1105.439470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_aborted -r 1 [ 1105.473878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_aborted -r 1 [ 1105.517302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_aborted -r 1 [ 1105.554289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_aborted -r 1 [ 1105.585255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_aborted -r 1 [ 1105.610577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_aborted -r 1 [ 1105.661939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_aborted -r 1 [ 1105.695492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1105.714212] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1105.719875] LNet: Skipped 2 previous similar messages [ 1105.919619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.944989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1105.970879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.001673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.031915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.069984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1106.100434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1106.161663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.184811] LNet: 60396:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1106.210383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.247648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.286962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.314547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.351362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.400937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.466076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1107.059952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_no_route -r 1 [ 1107.112135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_no_route -r 1 [ 1107.170358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_no_route -r 1 [ 1107.208257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_no_route -r 1 [ 1107.264509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_no_route -r 1 [ 1107.340628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_no_route -r 1 [ 1107.381648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_no_route -r 1 [ 1107.413656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_no_route -r 1 [ 1107.443394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1107.696603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.741245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1107.777710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.810275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.838980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.876268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.907931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.939929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.991491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.025327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.061354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.096291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.147350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.174618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.209488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1108.616578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_timeout -r 1 [ 1108.650725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_timeout -r 1 [ 1108.677690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_timeout -r 1 [ 1108.704725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_timeout -r 1 [ 1108.733174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_timeout -r 1 [ 1108.779979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_timeout -r 1 [ 1108.819650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_timeout -r 1 [ 1108.852696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_timeout -r 1 [ 1108.887882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1108.909106] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1108.917857] LNet: Skipped 5 previous similar messages [ 1109.113754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.164978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.216600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.268465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.306857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.363278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.420915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.500302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.548228] LNet: 60638:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1109.558229] LNet: 60638:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1109.596645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.648817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.685650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.711906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.733569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.762611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.793398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1110.328994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_error -r 1 [ 1110.367573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_error -r 1 [ 1110.409709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_error -r 1 [ 1110.455626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_error -r 1 [ 1110.490907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_error -r 1 [ 1110.534550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_error -r 1 [ 1110.565956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_error -r 1 [ 1110.615812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_error -r 1 [ 1110.648805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1110.885405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.927320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1110.964386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1111.002442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1111.048988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1111.073140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1111.100627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1111.126954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1111.167680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1111.222717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1111.288218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1111.333839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1111.376420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1111.415946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.454602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1116.325820] LNet: 60836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1116.330427] LNetError: 60836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1116.339255] LNet: Removed LNI 192.168.206.30@tcp [ 1117.363947] LNet: Removed LNI 192.168.206.30@tcp1 [ 1117.722737] Key type .llcrypt unregistered [ 1117.730837] Key type ._llcrypt unregistered [ 1134.334625] Key type ._llcrypt registered [ 1134.336930] Key type .llcrypt registered [ 1135.372808] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 19:50:18 (1784937018) [ 1136.500344] Key type .llcrypt unregistered [ 1136.502430] Key type ._llcrypt unregistered [ 1142.870386] Key type ._llcrypt registered [ 1142.872207] Key type .llcrypt registered [ 1143.057836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1143.067325] alg: No test for adler32 (adler32-zlib) [ 1144.024914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1144.072736] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1151.671350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1152.491209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_dropped -r 1 [ 1152.544974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_dropped -r 1 [ 1152.603879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1152.633373] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1152.903721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1152.972929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1153.004804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1153.052365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1153.114464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1153.157831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1153.202688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1153.239616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1153.264389] LNet: 62086:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1153.283858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1153.338916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1153.377710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1153.431561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1153.477834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1153.522066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1153.572826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1154.198883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_error -r 1 [ 1154.285395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_error -r 1 [ 1154.328881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1154.344850] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1154.539665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.581355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.625390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.678701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.721097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.743751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.786571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1154.817443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1154.841839] LNet: 62195:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1154.876295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1154.929348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1154.983466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.016367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.049386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.088710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.135334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1155.664244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_timeout -r 1 [ 1155.692837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_timeout -r 1 [ 1155.719579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1155.734434] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1155.940691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1155.979684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1156.044245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1156.092224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1156.131041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1156.158845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1156.184741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1156.221830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1156.244623] LNet: 62304:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1156.260525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1156.292995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1156.327803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1156.358576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1156.422815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1156.461963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1156.496798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1156.753561] LNet: 62345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.761332] LNetError: 62345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.782757] LNet: Removed LNI 192.168.206.30@tcp [ 1157.132960] Key type .llcrypt unregistered [ 1157.135703] Key type ._llcrypt unregistered [ 1174.191551] Key type ._llcrypt registered [ 1174.196896] Key type .llcrypt registered [ 1175.662544] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 19:50:58 (1784937058) [ 1176.763782] Key type .llcrypt unregistered [ 1176.771345] Key type ._llcrypt unregistered [ 1182.693643] Key type ._llcrypt registered [ 1182.696388] Key type .llcrypt registered [ 1182.939541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1182.955953] alg: No test for adler32 (adler32-zlib) [ 1183.904922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1183.931805] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1191.553057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1199.872861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1199.909540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1199.942308] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1201.254536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_dropped -r 1 [ 1201.305941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e remote_dropped -r 1 [ 1201.356553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_dropped -r 1 [ 1201.399068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e remote_dropped -r 1 [ 1201.451176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e remote_dropped -r 1 [ 1201.491269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e remote_dropped -r 1 [ 1201.524909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e remote_dropped -r 1 [ 1201.560599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e remote_dropped -r 1 [ 1201.603945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1201.617680] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1201.622581] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.130@tcp [ 1201.797943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1201.824508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1201.850975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1201.893930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1201.914353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1201.937904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1201.967600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1201.989259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1202.000311] LNet: 63813:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1202.022490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1202.050253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1202.095717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1202.147880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1202.187732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1202.239541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1202.296852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1202.364210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1202.869960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_error -r 1 [ 1202.901446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e remote_error -r 1 [ 1202.923465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_error -r 1 [ 1202.948817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e remote_error -r 1 [ 1202.979922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e remote_error -r 1 [ 1203.012747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e remote_error -r 1 [ 1203.046689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e remote_error -r 1 [ 1203.085324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e remote_error -r 1 [ 1203.115783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1203.140466] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1203.146483] LNet: Skipped 2 previous similar messages [ 1203.326925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1203.352607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1203.378625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1203.411355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1203.434896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1203.467329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1203.490779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1203.511089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1203.524417] LNet: 63937:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1203.541895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1203.573666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1203.601339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1203.633288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1203.667718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1203.706153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1203.743775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1203.785141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1204.257040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_timeout -r 1 [ 1204.287795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e remote_timeout -r 1 [ 1204.315957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_timeout -r 1 [ 1204.357291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e remote_timeout -r 1 [ 1204.382535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e remote_timeout -r 1 [ 1204.413519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e remote_timeout -r 1 [ 1204.461740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e remote_timeout -r 1 [ 1204.498657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e remote_timeout -r 1 [ 1204.528846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1204.547580] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1204.739714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1204.777138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1204.809618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1204.840724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1204.869128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1204.892299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1204.924747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1204.963880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1204.987300] LNet: 64060:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1205.005166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1205.047908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1205.075493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1205.116265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1205.144891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1205.169867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1205.204419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1205.245728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1209.929817] LNet: 64138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.934256] LNetError: 64138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.941403] LNet: Removed LNI 192.168.206.30@tcp [ 1209.949471] LNet: Removed LNI 192.168.206.30@tcp1 [ 1210.274542] Key type .llcrypt unregistered [ 1210.284219] Key type ._llcrypt unregistered [ 1227.097912] Key type ._llcrypt registered [ 1227.099940] Key type .llcrypt registered [ 1228.432673] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 19:51:51 (1784937111) [ 1230.098317] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1232.058975] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 19:51:54 (1784937114) [ 1233.252478] Key type .llcrypt unregistered [ 1233.255457] Key type ._llcrypt unregistered [ 1240.658332] Key type ._llcrypt registered [ 1240.663486] Key type .llcrypt registered [ 1240.988901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1241.015713] alg: No test for adler32 (adler32-zlib) [ 1242.092793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1242.136569] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1251.355993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1252.124061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e network_timeout -r 1 [ 1252.212379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e network_timeout -r 1 [ 1252.337325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1252.389061] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1252.798777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1252.885079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1252.949750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1253.014343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1253.077473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1253.150981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1253.217439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1253.274361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1253.309040] LNet: 65683:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1253.344076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1253.408779] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1253.415406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1253.443106] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1253.518035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1253.579601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1253.635474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1253.675345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1254.104815] LNet: 65722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1254.118477] LNetError: 65722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1255.148161] LNet: Removed LNI 192.168.206.30@tcp [ 1256.154684] Key type .llcrypt unregistered [ 1256.160397] Key type ._llcrypt unregistered [ 1276.187225] Key type ._llcrypt registered [ 1276.190383] Key type .llcrypt registered [ 1276.545610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1276.570591] alg: No test for adler32 (adler32-zlib) [ 1277.653275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1277.700525] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1285.304640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1294.615662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1294.681173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1294.720349] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1296.373803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e network_timeout -r 1 [ 1296.412856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e network_timeout -r 1 [ 1296.448227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e network_timeout -r 1 [ 1296.520307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e network_timeout -r 1 [ 1296.582235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e network_timeout -r 1 [ 1296.632439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e network_timeout -r 1 [ 1296.671538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e network_timeout -r 1 [ 1296.721684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e network_timeout -r 1 [ 1296.788514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1296.809785] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1297.098452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1297.129466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1297.156770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1297.183294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1297.189329] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1297.201176] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.130@tcp [ 1297.218132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1297.265027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1297.324442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1297.404203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1297.419337] LNet: 66602:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1297.443946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1297.477402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1297.523622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1297.563828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1297.602190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1297.644966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1302.264522] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1302.274615] LNet: Skipped 2 previous similar messages [ 1302.570517] LNet: 66676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1302.582245] LNetError: 66676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1303.584193] LNet: Removed LNI 192.168.206.30@tcp [ 1303.586715] LNetError: 66676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1303.603740] LNet: Removed LNI 192.168.206.30@tcp1 [ 1303.992564] Key type .llcrypt unregistered [ 1303.994569] Key type ._llcrypt unregistered [ 1320.029346] Key type ._llcrypt registered [ 1320.033826] Key type .llcrypt registered [ 1321.496112] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 19:53:24 (1784937204) [ 1321.822155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1321.850795] alg: No test for adler32 (adler32-zlib) [ 1322.850501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1322.954654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1323.005202] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1324.030515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1324.082354] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1325.433469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 1325.550813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1325.607821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1325.668895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1325.709688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1325.736986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1325.772084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.30@tcp [ 1325.796966] LNetError: 67776:0:(api-ni.c:4474:lnet_ni_set_healthv()) manually adding local NI 192.168.206.30@tcp to recovery [ 1325.984306] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1325.984442] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1327.009066] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1328.394532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1331.105944] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1332.164273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1332.230730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1332.337318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1332.410426] LNet: 67873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1332.424605] LNetError: 67873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.435982] LNet: Removed LNI 192.168.206.30@tcp [ 1332.449162] LNet: Removed LNI 192.168.206.30@tcp1 [ 1332.529760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1332.658933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1332.714197] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1333.602474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1333.635930] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1334.462939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 1334.503214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1334.595193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1334.673244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1334.725625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1334.777113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1334.807942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1334.858408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.30@tcp [ 1334.898439] LNetError: 68146:0:(api-ni.c:4474:lnet_ni_set_healthv()) manually adding local NI 192.168.206.30@tcp to recovery [ 1335.649161] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1338.344581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1341.792418] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1341.795660] LNet: Skipped 1 previous similar message [ 1346.256715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1346.288782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1346.333890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1353.209149] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 19:53:56 (1784937236) [ 1353.392676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1353.423681] LNet: 68863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1353.430252] LNetError: 68863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1353.439111] LNetError: 68863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1353.445258] LNet: Removed LNI 192.168.206.30@tcp [ 1353.456158] LNet: Removed LNI 192.168.206.30@tcp1 [ 1353.493549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1353.567936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1353.603248] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1354.379936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1354.404841] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1355.315871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 1355.352744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1355.370594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1355.393680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1355.416419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1355.499023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1355.534264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.30@tcp --health 0 [ 1355.552133] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.30@tcp [ 1355.613535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1356.578944] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1356.586081] LNet: Skipped 1 previous similar message [ 1371.323749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1371.652348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1371.723289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1371.773182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.30@tcp --health 500 [ 1372.005926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1379.204549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1379.308519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1379.347679] LNet: 69355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1379.367975] LNetError: 69355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1379.383175] LNetError: 69355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1379.412179] LNet: Removed LNI 192.168.206.30@tcp [ 1379.431609] LNet: Removed LNI 192.168.206.30@tcp1 [ 1379.480130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.557682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1379.624944] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1380.517469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1380.545360] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1381.850631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 1381.904976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1381.981091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1382.042524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1382.119530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1382.170666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1382.205319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1382.245637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.30@tcp --health 0 [ 1383.648369] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1383.655549] LNet: Skipped 3 previous similar messages [ 1384.746562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1395.000881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1395.029084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1395.072464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1401.827929] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 19:54:44 (1784937284) [ 1402.514900] LNet: 70412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1402.520814] LNetError: 70412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1402.527485] LNetError: 70412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1402.531970] LNet: Removed LNI 192.168.206.30@tcp [ 1402.541159] LNet: Removed LNI 192.168.206.30@tcp1 [ 1403.338908] Key type .llcrypt unregistered [ 1403.341540] Key type ._llcrypt unregistered [ 1410.495574] Key type ._llcrypt registered [ 1410.499513] Key type .llcrypt registered [ 1410.722848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1410.745398] alg: No test for adler32 (adler32-zlib) [ 1411.726915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1411.765403] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1420.919948] Lustre: DEBUG MARKER: Initial discovery [ 1421.038827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.130@tcp [ 1423.080932] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1423.385693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1423.444436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.130@tcp [ 1423.491548] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1423.524526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1423.582072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.130@tcp -r 1 -m GET -l 3 [ 1423.634436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.130@tcp -r 1 -m GET -e local_error [ 1423.684634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.130@tcp -r 1 -m PUT -l 6 [ 1424.096353] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1425.124626] Lustre: DEBUG MARKER: Force oleg630-client.virtnet to discover 192.168.206.130@tcp (in background) [ 1425.431544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.130@tcp [ 1426.614252] Lustre: DEBUG MARKER: Force oleg630-server to discover 192.168.206.30@tcp [ 1428.523205] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1429.368485] Lustre: DEBUG MARKER: Wait for 71177 [ 1430.930550] Lustre: DEBUG MARKER: Finished wait on 71177 [ 1431.237716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1431.281304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1431.282777] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1431.636751] LNet: 71624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.649973] LNetError: 71624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.679066] LNet: Removed LNI 192.168.206.30@tcp [ 1432.098642] Key type .llcrypt unregistered [ 1432.103230] Key type ._llcrypt unregistered [ 1450.619340] Key type ._llcrypt registered [ 1450.621536] Key type .llcrypt registered [ 1451.879676] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 19:55:34 (1784937334) [ 1452.326324] Key type .llcrypt unregistered [ 1452.331698] Key type ._llcrypt unregistered [ 1458.168419] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1458.225228] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1458.231521] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1458.262451] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1460.994877] Key type ._llcrypt registered [ 1460.997578] Key type .llcrypt registered [ 1461.205449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1461.216590] alg: No test for adler32 (adler32-zlib) [ 1462.232770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.305822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1462.358555] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1463.331698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1463.384791] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1472.759508] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 19:55:55 (1784937355) [ 1472.881110] LNet: 73695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1472.894994] LNetError: 73695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1472.903289] LNet: Removed LNI 10.1.2.3@tcp [ 1472.925328] LNet: Removed LNI 192.168.206.30@tcp [ 1473.619661] Key type .llcrypt unregistered [ 1473.622304] Key type ._llcrypt unregistered [ 1481.289298] Key type ._llcrypt registered [ 1481.295258] Key type .llcrypt registered [ 1481.619822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1481.647899] alg: No test for adler32 (adler32-zlib) [ 1482.699801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1482.815984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1482.872225] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1483.998756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1484.014527] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1485.502621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.30@tcp [ 1486.560248] LNetError: Unexpected error -22 connecting to 192.168.206.30@tcp at host 192.168.206.30:988 [ 1493.086638] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 19:56:15 (1784937375) [ 1493.372853] LNet: 74831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.384064] LNetError: 74831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1493.397096] LNet: Removed LNI 10.1.2.3@tcp [ 1493.432715] LNet: Removed LNI 192.168.206.30@tcp [ 1494.197662] Key type .llcrypt unregistered [ 1494.200744] Key type ._llcrypt unregistered [ 1501.479528] Key type ._llcrypt registered [ 1501.484055] Key type .llcrypt registered [ 1501.829349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1501.869288] alg: No test for adler32 (adler32-zlib) [ 1502.946523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1503.078247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1503.119289] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1503.853723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1503.888930] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 1504.762670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.30@tcp1 --nid 192.168.206.30@tcp2 [ 1508.700631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp1 192.168.206.30@tcp2 [ 1508.734705] LNetError: 75667:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.30@tcp1 to 192.168.206.30@tcp2 - Route table may be misconfigured [ 1508.778671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp2 192.168.206.30@tcp1 [ 1515.880597] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 19:56:38 (1784937398) [ 1516.074769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.107487] LNet: 76241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1516.111960] LNetError: 76241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1516.116206] LNet: Removed LNI 192.168.206.30@tcp1 [ 1517.158820] LNet: Removed LNI 192.168.206.30@tcp2 [ 1517.199792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.291719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1517.319111] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1518.088372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1518.105488] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1518.736768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 1518.808174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.30@tcp [ 1518.889083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp1 192.168.206.30@tcp1 [ 1518.922878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.30@tcp -d 192.168.206.30@tcp -e network_timeout [ 1518.972885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -e network_timeout [ 1519.005749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -e network_timeout [ 1519.042263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -e network_timeout [ 1519.081293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1519.103481] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1519.129317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1519.237453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1519.335420] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1519.358041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1525.265451] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 19:56:47 (1784937407) [ 1525.383589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1525.412980] LNet: 77095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.422815] LNetError: 77095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1525.430566] LNetError: 77095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1525.437354] LNet: Removed LNI 192.168.206.30@tcp [ 1526.504491] LNet: Removed LNI 192.168.206.30@tcp1 [ 1526.527446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1526.610288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1526.822571] LNet: 77141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1527.131878] Key type .llcrypt unregistered [ 1527.134911] Key type ._llcrypt unregistered [ 1538.612996] Key type ._llcrypt registered [ 1538.620508] Key type .llcrypt registered [ 1540.002462] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 19:57:02 (1784937422) [ 1540.619063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1540.634383] alg: No test for adler32 (adler32-zlib) [ 1541.643847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.262744] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1545.382282] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 19:57:07 (1784937427) [ 1545.486680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1545.522513] LNet: 78232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.556416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.640639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1545.689627] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1546.434934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1546.475580] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1547.390541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1547.453176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp1 [ 1547.536644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp1 [ 1554.309548] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 19:57:16 (1784937436) [ 1556.131384] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg630-server [ 1557.646317] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 19:57:20 (1784937440) [ 1559.469425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg630-server [ 1561.240836] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 19:57:23 (1784937443) [ 1563.263797] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg630-server [ 1565.238289] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 19:57:27 (1784937447) [ 1567.301336] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg630-server [ 1569.041861] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 19:57:31 (1784937451) [ 1570.710273] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg630-server [ 1572.328780] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 19:57:35 (1784937455) [ 1574.186450] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg630-server [ 1576.477483] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 19:57:38 (1784937458) [ 1578.293227] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg630-server [ 1580.103584] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 19:57:43 (1784937463) [ 1581.882815] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg630-server [ 1583.732388] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 19:57:46 (1784937466) [ 1585.372225] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg630-server [ 1587.299502] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 19:57:49 (1784937469) [ 1587.462548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.495305] LNet: 82115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.506641] LNetError: 82115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.576461] LNet: Removed LNI 192.168.206.30@tcp [ 1588.579857] LNetError: 82115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.618572] LNet: Removed LNI 192.168.206.30@tcp1 [ 1588.659634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.719321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1588.749624] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1589.625629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1589.724512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1589.945808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.982711] LNet: 82264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.995147] LNetError: 82264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1590.022074] LNet: Removed LNI 192.168.206.30@tcp [ 1590.055854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.126707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1590.156688] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1591.051433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1591.260076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.295061] LNet: Removed LNI 192.168.206.30@tcp [ 1592.341850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.429507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1592.461442] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1593.174908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1593.269042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1593.556773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.612112] LNet: 82557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.627224] LNet: 82557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1593.639410] LNetError: 82557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1593.642980] LNetError: 82557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1594.664770] LNet: Removed LNI 192.168.206.30@tcp [ 1594.730061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.818667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1594.856745] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1595.833747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1596.021721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.097756] LNet: Removed LNI 192.168.206.30@tcp [ 1597.152022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.232639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1597.260086] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1598.199495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1598.276548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1598.484611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.509394] LNet: 82850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.516266] LNet: 82850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1598.521911] LNetError: 82850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1598.531373] LNetError: 82850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1598.554644] LNet: Removed LNI 192.168.206.30@tcp [ 1598.587821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.663131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1598.719502] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1599.504909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1599.824631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.936267] LNet: Removed LNI 192.168.206.30@tcp [ 1600.999477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.053437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1601.089516] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1602.257521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1602.338647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1602.714715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.829844] LNet: Removed LNI 192.168.206.30@tcp [ 1603.861972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.981720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1604.042348] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1604.825246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1606.363847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.475467] LNet: Removed LNI 192.168.206.30@tcp [ 1607.510838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.565437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1607.584982] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1608.650255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1608.775117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1609.037449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.077753] LNet: 83445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.081690] LNet: 83445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1609.100309] LNetError: 83445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1609.107982] LNetError: 83445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1609.123480] LNet: Removed LNI 192.168.206.30@tcp [ 1609.178504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.335916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1609.401372] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1610.429945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1610.857271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.951621] LNet: Removed LNI 192.168.206.30@tcp [ 1612.016307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.075257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1612.107549] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1613.096339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1613.261313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1614.770505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.861119] LNet: Removed LNI 192.168.206.30@tcp [ 1615.930565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.032601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1616.055914] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1617.402521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1618.988106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.070865] LNet: Removed LNI 192.168.206.30@tcp [ 1620.133950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.242916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1620.282947] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1621.211613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1621.277776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.316208] LNet: Removed LNI 192.168.206.30@tcp [ 1621.355848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.412911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1621.439174] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1622.364676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1622.461544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.528208] LNet: Removed LNI 192.168.206.30@tcp [ 1622.541637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.607098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1622.695236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.740708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.807822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1622.892881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.936627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.984146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1623.076869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.120709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.164692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1623.186411] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1624.063692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.085906] LNet: Removed LNI 192.168.206.30@tcp [ 1624.107865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.183184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1631.095487] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 19:58:33 (1784937513) [ 1631.194760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.217481] LNet: 84918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.220696] LNet: 84918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1631.242245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.289539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1631.310927] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/180] [ 1632.604934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1632.635396] LNetError: 85061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1632.641754] LNetError: 85061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1632.655894] LNet: Removed LNI 192.168.206.30@tcp231 [ 1632.672810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.755665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1632.783120] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/180] [ 1632.792864] LNet: 85069:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1633.416600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1633.472612] LNet: Removed LNI 192.168.206.30@tcp231 [ 1633.493829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1633.542432] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/180] [ 1634.659714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.702718] LNet: Removed LNI 192.168.206.30@tcp231 [ 1634.740811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.845209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1634.889534] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/0] [ 1634.898403] LNet: 85222:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1635.398234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1635.464116] LNet: Removed LNI 192.168.206.30@tcp231 [ 1635.488466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1635.521598] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/0] [ 1636.601348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.648107] LNet: Removed LNI 192.168.206.30@tcp231 [ 1636.693500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.826612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1636.875273] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/60] [ 1636.887282] LNet: 85373:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1637.237605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1637.276286] LNet: Removed LNI 192.168.206.30@tcp231 [ 1637.312538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1637.338137] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/60] [ 1638.365501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.383033] LNet: Removed LNI 192.168.206.30@tcp231 [ 1638.402470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.459228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1638.494327] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/180] [ 1638.773962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1638.798570] LNet: Removed LNI 192.168.206.30@tcp231 [ 1638.814852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1638.834476] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/180] [ 1646.420778] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 19:58:49 (1784937529) [ 1646.593933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.639721] LNet: Removed LNI 192.168.206.30@tcp231 [ 1646.669223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.760230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1646.811647] LNet: Added LNI 192.168.206.30@tcp232 [8/256/0/180] [ 1647.516618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1647.596451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1647.653882] LNet: Removed LNI 192.168.206.30@tcp232 [ 1647.671980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1647.711302] LNet: Added LNI 192.168.206.30@tcp232 [8/256/0/180] [ 1648.890541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1648.977899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1649.017764] LNet: Removed LNI 192.168.206.30@tcp232 [ 1649.044939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1649.075758] LNet: Added LNI 192.168.206.30@tcp232 [8/256/0/180] [ 1649.580750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1649.614505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1649.647457] LNet: Removed LNI 192.168.206.30@tcp232 [ 1649.668914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1649.710174] LNet: Added LNI 192.168.206.30@tcp232 [8/256/0/180] [ 1650.499330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1650.551930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1650.609929] LNet: Removed LNI 192.168.206.30@tcp232 [ 1650.633908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1657.507321] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 19:59:00 (1784937540) [ 1658.854523] Key type .llcrypt unregistered [ 1658.860029] Key type ._llcrypt unregistered [ 1666.098234] Key type ._llcrypt registered [ 1666.101170] Key type .llcrypt registered [ 1666.328159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1666.351816] alg: No test for adler32 (adler32-zlib) [ 1667.310141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1667.352193] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1674.967145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1683.912232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.967859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1684.008328] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1685.695761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_interrupt -r 2 [ 1685.721745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_interrupt -r 2 [ 1685.747915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_interrupt -r 2 [ 1685.774676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_interrupt -r 2 [ 1685.801779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_interrupt -r 2 [ 1685.831756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_interrupt -r 2 [ 1685.863251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_interrupt -r 2 [ 1685.902195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_interrupt -r 2 [ 1685.950759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1685.989334] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1686.002016] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1686.406800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1686.449691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1686.510864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1686.551079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1686.614160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1686.650414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1686.701885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1686.742537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1686.765731] LNet: 88027:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1686.794263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1686.837665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1686.895317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1686.943586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1687.014763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1687.056806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1687.743726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_dropped -r 2 [ 1687.804354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_dropped -r 2 [ 1687.878947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_dropped -r 2 [ 1687.941849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_dropped -r 2 [ 1687.989629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_dropped -r 2 [ 1688.035332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_dropped -r 2 [ 1688.104478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_dropped -r 2 [ 1688.163430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_dropped -r 2 [ 1688.224856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1688.260526] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1688.270817] LNet: Skipped 1 previous similar message [ 1688.617389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1688.652304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1688.687726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1688.758715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1688.810239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1688.843585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1688.892342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1688.944725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1688.970242] LNet: 88147:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1689.001089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1689.033938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1689.075519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1689.127366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1689.175557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1689.221607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1689.894679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_aborted -r 2 [ 1689.941858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_aborted -r 2 [ 1689.980863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_aborted -r 2 [ 1690.046372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_aborted -r 2 [ 1690.106816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_aborted -r 2 [ 1690.169406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_aborted -r 2 [ 1690.196487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_aborted -r 2 [ 1690.236919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_aborted -r 2 [ 1690.281505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1690.307982] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1690.569275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1690.589576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1690.609971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1690.632781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1690.656120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1690.679902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1690.715116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1690.738300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1690.753960] LNet: 88268:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1690.770835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1690.793816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1690.814272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1690.838826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1690.862749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1690.892248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1691.452528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_no_route -r 2 [ 1691.492239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_no_route -r 2 [ 1691.542971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_no_route -r 2 [ 1691.589945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_no_route -r 2 [ 1691.634821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_no_route -r 2 [ 1691.685609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_no_route -r 2 [ 1691.734805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_no_route -r 2 [ 1691.814792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_no_route -r 2 [ 1691.884997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1692.165817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1692.207404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1692.248860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1692.302467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1692.360860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1692.407957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1692.441257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1692.481220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1692.522922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1692.567354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1692.596987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1692.628209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1692.661294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1692.711852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1693.447790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_timeout -r 2 [ 1693.484040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_timeout -r 2 [ 1693.518898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_timeout -r 2 [ 1693.566793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_timeout -r 2 [ 1693.612922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_timeout -r 2 [ 1693.644235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_timeout -r 2 [ 1693.719299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_timeout -r 2 [ 1693.765582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_timeout -r 2 [ 1693.821178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1693.847331] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1693.852228] LNet: Skipped 1 previous similar message [ 1694.178915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1694.219883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1694.247618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1694.296036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1694.338689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1694.389817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1694.436963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1694.479554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1694.511805] LNet: 88508:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1694.524805] LNet: 88508:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1694.554405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1694.593186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1694.627814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1694.656268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1694.686478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1694.734221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1695.657950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_dropped -r 2 [ 1695.729557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e remote_dropped -r 2 [ 1695.823375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_dropped -r 2 [ 1695.876253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e remote_dropped -r 2 [ 1695.970915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e remote_dropped -r 2 [ 1696.051655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e remote_dropped -r 2 [ 1696.165062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e remote_dropped -r 2 [ 1696.251529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e remote_dropped -r 2 [ 1696.308165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1696.332794] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.130@tcp [ 1696.548933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1696.587239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1696.659200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1696.685436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1696.714734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1696.749659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1696.784840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1696.814927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1696.850835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1696.880699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1696.927235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1696.978995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1697.016900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1697.081286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1702.725825] LNet: 88703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.734475] LNetError: 88703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.747488] LNet: Removed LNI 192.168.206.30@tcp [ 1702.770510] LNet: Removed LNI 192.168.206.30@tcp1 [ 1703.343570] Key type .llcrypt unregistered [ 1703.347956] Key type ._llcrypt unregistered [ 1722.629821] Key type ._llcrypt registered [ 1722.631951] Key type .llcrypt registered [ 1723.865556] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 20:00:06 (1784937606) [ 1724.400596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1724.412918] alg: No test for adler32 (adler32-zlib) [ 1725.296979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.348279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.372229] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1726.222745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1726.258891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.30@tcp [ 1726.281979] LNetError: 89684:0:(api-ni.c:4474:lnet_ni_set_healthv()) manually adding local NI 192.168.206.30@tcp to recovery [ 1726.364761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.30@tcp [ 1726.377404] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1726.497055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1732.591809] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 20:00:15 (1784937615) [ 1732.703267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1732.726372] LNet: 90272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1732.738827] LNetError: 90272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.761685] LNet: Removed LNI 192.168.206.30@tcp [ 1732.783851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.843662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1732.869685] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1733.597431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1733.629468] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1734.537893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.30@tcp [ 1734.550636] LNetError: 90540:0:(api-ni.c:4474:lnet_ni_set_healthv()) manually adding local NI 192.168.206.30@tcp to recovery [ 1734.882154] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1735.752888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.30@tcp [ 1744.451260] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 20:00:27 (1784937627) [ 1746.153796] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg630-server [ 1747.899617] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 20:00:30 (1784937630) [ 1749.831431] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg630-server [ 1751.483874] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 20:00:34 (1784937634) [ 1753.531869] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg630-server [ 1755.526994] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 20:00:38 (1784937638) [ 1755.639055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.698290] LNet: 92165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.704219] LNetError: 92165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.726579] LNet: Removed LNI 192.168.206.30@tcp [ 1755.749311] LNet: Removed LNI 192.168.206.30@tcp1 [ 1755.805766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.888059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1755.928309] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1756.482694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1756.537369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1756.570160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1756.617171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1756.656700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1756.687735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1762.637296] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 20:00:45 (1784937645) [ 1762.720911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.775532] LNet: 92888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1762.786464] LNetError: 92888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.804687] LNetError: 92888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1762.847476] LNet: Removed LNI 192.168.206.30@tcp [ 1762.893188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.965802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1763.003517] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1770.878235] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 20:00:53 (1784937653) [ 1771.099343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.147429] LNet: 93593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.154176] LNetError: 93593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.174673] LNet: Removed LNI 192.168.206.30@tcp [ 1771.208137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.289608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1771.316343] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1778.991180] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 20:01:01 (1784937661) [ 1780.683626] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1782.122730] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 20:01:05 (1784937665) [ 1782.864461] LNet: 94648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1782.876939] LNetError: 94648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.915153] LNet: Removed LNI 192.168.206.30@tcp [ 1783.593745] Key type .llcrypt unregistered [ 1783.597955] Key type ._llcrypt unregistered [ 1789.258142] Key type ._llcrypt registered [ 1789.260257] Key type .llcrypt registered [ 1789.449475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1789.463496] alg: No test for adler32 (adler32-zlib) [ 1790.342586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1790.372815] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1797.778218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1802.936795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.130@tcp [ 1803.287151] LNet: 95065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.293523] LNetError: 95065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1803.308909] LNet: Removed LNI 192.168.206.30@tcp [ 1803.736398] Key type .llcrypt unregistered [ 1803.740129] Key type ._llcrypt unregistered [ 1816.000887] Key type ._llcrypt registered [ 1816.003625] Key type .llcrypt registered [ 1817.568477] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1819.538878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1821.499554] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 20:01:44 (1784937704) [ 1823.365146] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1825.414287] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 20:01:47 (1784937707) [ 1827.486677] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg630-server [ 1829.948230] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 20:01:52 (1784937712) [ 1831.826705] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg630-server [ 1833.290832] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 20:01:56 (1784937716) [ 1834.867908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1834.914030] alg: No test for adler32 (adler32-zlib) [ 1835.990175] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 1836.224124] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1837.931263] Key type lgssc registered [ 1839.056373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1858.604477] Key type lgssc unregistered [ 1858.818565] LNet: 98240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.827328] LNetError: 98240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1858.843290] LNet: Removed LNI 192.168.206.30@tcp [ 1859.279588] Key type .llcrypt unregistered [ 1859.286808] Key type ._llcrypt unregistered [ 1876.501248] Key type ._llcrypt registered [ 1876.503204] Key type .llcrypt registered [ 1878.044354] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 20:02:40 (1784937760) [ 1878.488636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1878.525511] alg: No test for adler32 (adler32-zlib) [ 1879.452773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1879.566801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1879.597813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1879.655308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1879.713363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1887.937748] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 20:02:50 (1784937770) [ 1888.027299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1888.064789] LNet: 99651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1888.079734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1888.153833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1888.182737] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1888.983769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1889.018502] LNet: 99799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.027940] LNetError: 99799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.044823] LNet: Removed LNI 192.168.206.30@tcp [ 1889.071402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1889.163824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1889.198137] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1889.756850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1889.773429] LNetError: 99828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.790253] LNet: Removed LNI 192.168.206.30@tcp [ 1889.824810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1889.862723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1889.880574] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1896.590242] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 20:02:59 (1784937779) [ 1896.787767] LNet: 100418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.793420] LNet: 100418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1896.802711] LNetError: 100418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.815584] LNet: Removed LNI 192.168.206.30@tcp [ 1897.415535] Key type .llcrypt unregistered [ 1897.418927] Key type ._llcrypt unregistered [ 1904.073469] Key type ._llcrypt registered [ 1904.076465] Key type .llcrypt registered [ 1904.382424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1904.421794] alg: No test for adler32 (adler32-zlib) [ 1905.438730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.561743] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1905.564142] LNet: *** cfs_fail_loc=e003, val=100*** [ 1906.211971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1906.245412] LNet: *** cfs_fail_loc=e003, val=100*** [ 1906.248706] LNet: Skipped 2 previous similar messages [ 1906.258297] Ping reply received with mapping: [ 1906.260743] found metadata reply NI[192.168.206.30@tcp] [ 1906.263170] 1e ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1906.266127] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1906.269404] 5a 5a 5a 5a 5a 5a 5a 5a [ 1906.465636] LNet: 100815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1906.474695] LNetError: 100815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1906.498659] LNet: Removed LNI 192.168.206.30@tcp [ 1907.231671] Key type .llcrypt unregistered [ 1907.235380] Key type ._llcrypt unregistered [ 1918.863427] Key type ._llcrypt registered [ 1918.865551] Key type .llcrypt registered [ 1920.270234] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 20:03:22 (1784937802) [ 1921.092573] Key type .llcrypt unregistered [ 1921.100060] Key type ._llcrypt unregistered [ 1927.565445] Key type ._llcrypt registered [ 1927.568652] Key type .llcrypt registered [ 1927.755586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1927.767932] alg: No test for adler32 (adler32-zlib) [ 1929.060657] LNetError: 101861:0:(api-ni.c:2851:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1929.572937] Key type .llcrypt unregistered [ 1929.578293] Key type ._llcrypt unregistered [ 1935.077549] Key type ._llcrypt registered [ 1935.079647] Key type .llcrypt registered [ 1936.292359] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 20:03:39 (1784937819) [ 1937.015781] Key type .llcrypt unregistered [ 1937.017507] Key type ._llcrypt unregistered [ 1942.311851] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1942.320158] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1944.992856] Key type ._llcrypt registered [ 1944.994686] Key type .llcrypt registered [ 1945.218605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1945.230834] alg: No test for adler32 (adler32-zlib) [ 1946.202334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.294346] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1955.630793] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 20:03:58 (1784937838) [ 1955.817086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1955.845581] LNet: 103641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1955.856185] LNetError: 103641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1955.869162] LNet: Removed LNI 10.1.2.3@tcp [ 1955.882488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1956.517778] Key type .llcrypt unregistered [ 1956.524440] Key type ._llcrypt unregistered [ 1963.071492] Key type ._llcrypt registered [ 1963.074600] Key type .llcrypt registered [ 1963.192262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1963.201877] alg: No test for adler32 (adler32-zlib) [ 1964.168262] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1964.222766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1964.377392] LNet: 103929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.395774] LNetError: 103929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1964.419129] LNet: Removed LNI 192.168.206.30@tcp [ 1964.979629] Key type .llcrypt unregistered [ 1964.981374] Key type ._llcrypt unregistered [ 1975.540571] Key type ._llcrypt registered [ 1975.544980] Key type .llcrypt registered [ 1977.098939] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 20:04:19 (1784937859) [ 1977.803487] Key type .llcrypt unregistered [ 1977.807616] Key type ._llcrypt unregistered [ 1984.524650] Key type ._llcrypt registered [ 1984.526551] Key type .llcrypt registered [ 1984.849111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1984.881525] alg: No test for adler32 (adler32-zlib) [ 1993.123682] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 20:04:35 (1784937875) [ 1993.365082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1993.458337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.563382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1993.599309] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1994.245419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1994.284801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1994.311655] LNet: Removed LNI 192.168.206.30@tcp [ 1994.565487] LNet: 105673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1994.937368] Key type .llcrypt unregistered [ 1994.942942] Key type ._llcrypt unregistered [ 2006.594348] Key type ._llcrypt registered [ 2006.597336] Key type .llcrypt registered [ 2008.112922] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 20:04:50 (1784937890) [ 2008.789216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2008.827366] alg: No test for adler32 (adler32-zlib) [ 2009.697299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2009.751108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2009.792792] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2011.231820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2011.327358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.30@tcp [ 2011.353946] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2011.358779] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2011.363328] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2011, 0, 0 [ 2011.371257] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2011.380294] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2011.386118] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2011.392912] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2011, 0, 0 [ 2011.399175] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2011.405972] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2011.409183] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2011.413420] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2011, 0, 2061 [ 2011.425540] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2011.431780] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2011.440007] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2011.445298] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2011, 0, 2061 [ 2011.453559] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2018.111253] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 20:05:01 (1784937901) [ 2018.489873] LNet: 107213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2018.496768] LNetError: 107213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.558359] LNet: Removed LNI 192.168.206.30@tcp [ 2020.247551] Key type .llcrypt unregistered [ 2020.251991] Key type ._llcrypt unregistered [ 2026.188504] Key type ._llcrypt registered [ 2026.192977] Key type .llcrypt registered [ 2026.361348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2026.379230] alg: No test for adler32 (adler32-zlib) [ 2027.300618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2027.325637] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2034.781907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 2043.283445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2043.325657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2043.371342] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 2044.140631] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2044.291340] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2044.299388] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2044.306981] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2046.659409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2046.675603] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2056.261717] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 20:05:39 (1784937939) [ 2056.476617] LNet: 108721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2056.488473] LNetError: 108721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2056.496099] LNet: Removed LNI 192.168.206.30@tcp [ 2057.504158] LNet: Removed LNI 192.168.206.30@tcp1 [ 2057.508313] LNetError: 108721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2057.519227] LNetError: 108721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2057.537620] LNet: Removed LNI 10.1.2.3@tcp99 [ 2058.156593] Key type .llcrypt unregistered [ 2058.158838] Key type ._llcrypt unregistered [ 2064.312107] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2064.316295] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2066.904207] Key type ._llcrypt registered [ 2066.908469] Key type .llcrypt registered [ 2067.084635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2067.097492] alg: No test for adler32 (adler32-zlib) [ 2068.001673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2068.074495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2068.097195] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2068.829218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2068.865792] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2070.800492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2070.832167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2070.882584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2070.911082] LNetError: 109479:0:(peer.c:2277:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2070.934782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2070.970502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2071.281218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2071.365827] LNet: 109500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2071.373507] LNetError: 109500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2071.379764] LNet: Removed LNI 10.1.2.3@tcp [ 2072.436301] LNet: Removed LNI 192.168.206.30@tcp [ 2073.179378] Key type .llcrypt unregistered [ 2073.184541] Key type ._llcrypt unregistered [ 2084.992713] Key type ._llcrypt registered [ 2084.998521] Key type .llcrypt registered [ 2086.333288] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 20:06:09 (1784937969) [ 2086.917119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2086.938695] alg: No test for adler32 (adler32-zlib) [ 2087.882927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2087.948224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2087.990486] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2088.786133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2088.874955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg630-client@tcp [ 2094.612557] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 20:06:17 (1784937977) [ 2094.705742] LNet: 111023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2094.709418] LNetError: 111023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2094.737369] LNet: Removed LNI 192.168.206.30@tcp [ 2095.488605] Key type .llcrypt unregistered [ 2095.493454] Key type ._llcrypt unregistered [ 2102.689492] Key type ._llcrypt registered [ 2102.697448] Key type .llcrypt registered [ 2102.990764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2102.999312] alg: No test for adler32 (adler32-zlib) [ 2103.940629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2103.975252] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2104.003072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2112.235271] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 20:06:34 (1784937994) [ 2114.357755] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2116.325069] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 20:06:38 (1784937998) [ 2117.976744] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2120.136721] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 20:06:42 (1784938002) [ 2120.263290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2120.296819] LNet: 112446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.302254] LNetError: 112446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2120.313465] LNet: Removed LNI 192.168.206.30@tcp [ 2120.333636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2120.385389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2120.415672] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2121.249056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2121.353934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2128.266753] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 20:06:51 (1784938011) [ 2128.633220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2128.672964] LNet: 113161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.686925] LNetError: 113161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2129.703626] LNet: Removed LNI 192.168.206.30@tcp [ 2129.741819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.847710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2129.943372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2130.411102] LNet: 113198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2130.882282] Key type .llcrypt unregistered [ 2130.887708] Key type ._llcrypt unregistered [ 2140.636151] Key type ._llcrypt registered [ 2140.638956] Key type .llcrypt registered [ 2141.810809] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 20:07:04 (1784938024) [ 2142.165339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2142.183192] alg: No test for adler32 (adler32-zlib) [ 2143.076515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.116404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2143.131209] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2143.677471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2143.722810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2149.237736] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 20:07:12 (1784938032) [ 2149.346460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2149.384733] LNet: 114711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2149.396808] LNetError: 114711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2150.442354] LNet: Removed LNI 192.168.206.30@tcp [ 2150.473186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.584963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2150.640994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2156.950771] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 20:07:19 (1784938039) [ 2157.171789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2157.207442] LNet: 115297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2157.226836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.284633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2157.702394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2157.749560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2157.832512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2158.238582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2158.297105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2158.395290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2158.424427] LNet: 115330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.449037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.519784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2158.937703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2158.989145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2159.058815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2159.397502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2159.453346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2165.199543] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 20:07:28 (1784938048) [ 2165.464546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2165.484437] LNet: 115930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.517364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.572794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2165.592297] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2166.153599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2166.172689] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2167.046988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2167.107695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2167.315625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2167.368450] LNetError: 116210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2167.378992] LNet: Removed LNI 192.168.206.30@tcp [ 2167.390927] LNet: Removed LNI 192.168.206.30@tcp2 [ 2167.424391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2167.522872] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2167.528495] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2167.533567] LNet: 116216:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2168.083859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2168.269946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2169.312249] LNet: Removed LNI 192.168.206.30@tcp [ 2169.314219] LNetError: 116253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2169.319479] LNetError: 116253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2169.342426] LNet: Removed LNI 192.168.206.30@tcp2 [ 2169.384734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2169.478210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2169.512601] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2170.226381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2170.241905] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2170.965219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2171.012956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2171.082895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2171.303954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2171.328086] LNet: 116537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2171.332682] LNet: 116537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2171.346466] LNetError: 116537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2171.355602] LNet: Removed LNI 192.168.206.30@tcp [ 2172.400340] LNet: Removed LNI 192.168.206.30@tcp2 [ 2172.449119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2172.571726] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2172.583331] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2172.600282] LNet: 116544:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2173.409498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2173.802836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2173.864668] LNet: Removed LNI 192.168.206.30@tcp [ 2174.887815] LNet: Removed LNI 192.168.206.30@tcp2 [ 2175.000948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2175.212188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2175.272080] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2176.109644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2176.126694] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2177.214463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2177.287863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2177.561331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2177.594854] LNetError: 116860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2177.600757] LNetError: 116860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2177.620781] LNet: Removed LNI 192.168.206.30@tcp [ 2177.644977] LNet: Removed LNI 192.168.206.30@tcp2 [ 2177.686661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2177.828789] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2177.844413] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2177.853527] LNet: 116866:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2178.967918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2179.251069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2180.320378] LNet: Removed LNI 192.168.206.30@tcp [ 2180.345595] LNet: Removed LNI 192.168.206.30@tcp2 [ 2180.383681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.462331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2180.501961] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2181.665346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2181.699205] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2183.364681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2183.423909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2183.547406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2183.919291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2183.967287] LNet: 117188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2183.986390] LNet: 117188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2185.056163] LNet: Removed LNI 192.168.206.30@tcp [ 2186.096195] LNet: Removed LNI 192.168.206.30@tcp2 [ 2186.143677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.248982] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2186.264687] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2186.274309] LNet: 117195:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2186.780688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2186.992590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2187.007208] LNetError: 117232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2187.010778] LNetError: 117232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2187.016400] LNet: Removed LNI 192.168.206.30@tcp [ 2188.074174] LNet: Removed LNI 192.168.206.30@tcp2 [ 2188.113117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2188.201885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2188.237629] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2189.301319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2189.355569] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2190.856262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2190.953569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2191.123953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2191.827788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2192.928167] LNet: Removed LNI 192.168.206.30@tcp [ 2192.952719] LNet: Removed LNI 192.168.206.30@tcp2 [ 2192.994395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.099975] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2193.111340] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2193.114396] LNet: 117534:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2193.810456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2194.384463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2195.424444] LNet: Removed LNI 192.168.206.30@tcp [ 2195.433737] LNet: Removed LNI 192.168.206.30@tcp2 [ 2195.493971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2195.544459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2195.566266] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2196.365938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2196.394330] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2197.258828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2197.315661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2197.357363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2197.418925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2197.773422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2198.817824] LNet: Removed LNI 192.168.206.30@tcp [ 2199.863767] LNet: Removed LNI 192.168.206.30@tcp2 [ 2199.943633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2200.042962] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2200.052780] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2200.783878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2209.118122] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2211.072819] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 20:08:13 (1784938093) [ 2211.181325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2211.228346] LNet: 118646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2211.236375] LNet: 118646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2211.253865] LNetError: 118646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2211.259783] LNetError: 118646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2211.273181] LNet: Removed LNI 192.168.206.30@tcp [ 2211.288142] LNet: Removed LNI 192.168.206.30@tcp2 [ 2211.313110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2211.677763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2211.735414] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2212.683443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2212.708898] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2213.787170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2213.816384] LNet: Removed LNI 10.1.2.3@tcp [ 2213.833130] LNet: Removed LNI 192.168.206.30@tcp [ 2213.871240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2213.983056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2214.032964] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2214.043920] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2214.624337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2214.657472] LNet: Removed LNI 192.168.206.30@tcp [ 2214.667113] LNet: Removed LNI 10.1.2.3@tcp [ 2222.869933] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 20:08:24 (1784938104) [ 2223.037483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2223.132433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2223.266194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2223.324708] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2224.320428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2224.373423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2224.407916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2224.456385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2224.513614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2224.585705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2224.632553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2224.730264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2224.765966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2224.827607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2224.893978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2225.092429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2234.663172] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 20:08:37 (1784938117) [ 2234.795692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2235.882362] LNet: Removed LNI 192.168.206.30@tcp [ 2235.972923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2236.210967] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2236.310894] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2236.317896] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2236.323062] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2238.797598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2238.827057] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2239.558810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2239.574753] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2241.795662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2241.858403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2241.947362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.30@tcp -m PUT -e local_timeout -r 1 [ 2242.025543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2242.087570] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2242.090858] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2242.336657] LNetError: 120329:0:(lib-move.c:4801:lnet_parse()) 192.168.206.30@tcp, src 192.168.206.30@tcp: Dropping PUT (error -108 looking up sender) [ 2243.371367] LNet: 120789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2243.386089] LNet: 120789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2243.397044] LNetError: 120789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2243.413233] LNetError: 120789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2243.432136] LNet: Removed LNI 10.1.2.3@tcp [ 2244.457874] LNet: Removed LNI 192.168.206.30@tcp [ 2244.843534] Key type .llcrypt unregistered [ 2244.846062] Key type ._llcrypt unregistered [ 2256.103809] Key type ._llcrypt registered [ 2256.109616] Key type .llcrypt registered [ 2257.509067] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 20:09:00 (1784938140) [ 2257.867538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2257.882843] alg: No test for adler32 (adler32-zlib) [ 2258.833987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2259.185556] LNet: 121640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2259.795560] Key type .llcrypt unregistered [ 2259.799939] Key type ._llcrypt unregistered [ 2266.584849] Key type ._llcrypt registered [ 2266.595231] Key type .llcrypt registered [ 2266.766867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2266.783762] alg: No test for adler32 (adler32-zlib) [ 2267.710880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2267.740527] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2275.646389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 2275.866847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2276.415584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.130@tcp [ 2276.749742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2276.782623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2276.823262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2276.863664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2276.910976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2276.951706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2276.964798] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.130@tcp [ 2276.987798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2277.034401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2277.063901] LNet: 122090:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2277.090912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2277.131744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2277.170569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2277.209536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2277.244632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2277.288161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2278.050526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2278.345528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2278.399498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2278.451883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2278.490162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2278.552187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2278.599573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2278.627212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2278.654726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2278.670926] LNet: 122207:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2278.690293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2278.712176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2278.735731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2278.769047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2278.814275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2278.851797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2279.452755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.130@tcp [ 2279.752497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2279.802281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2279.861340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2279.921830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2279.954680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2279.982703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2280.041791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2280.070430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2280.100169] LNet: 122324:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2280.124866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2280.162988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2280.188411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2280.221689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2280.263294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2280.312984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2281.062317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2281.340958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2281.395293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2281.444615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2281.481059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2281.527955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2281.573441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2281.622531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2281.657496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2281.688698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2281.726887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2281.753963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2281.791813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2281.842912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2281.880715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2282.202487] LNet: 122487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2282.209700] LNetError: 122487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2282.224529] LNet: Removed LNI 192.168.206.30@tcp [ 2282.683363] Key type .llcrypt unregistered [ 2282.694780] Key type ._llcrypt unregistered [ 2301.608342] Key type ._llcrypt registered [ 2301.610778] Key type .llcrypt registered [ 2303.127733] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 20:09:45 (1784938185) [ 2303.633554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2303.646390] alg: No test for adler32 (adler32-zlib) [ 2304.560492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.917585] LNet: 123375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.459780] Key type .llcrypt unregistered [ 2305.462602] Key type ._llcrypt unregistered [ 2312.627190] Key type ._llcrypt registered [ 2312.630475] Key type .llcrypt registered [ 2312.903724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2312.921161] alg: No test for adler32 (adler32-zlib) [ 2313.982973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2314.011587] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2321.627893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 2331.802875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.871512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2331.902820] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 2333.195008] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2334.142820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.130@tcp [ 2334.500750] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.130@tcp [ 2334.516704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2334.559492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2334.638651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2334.706580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2334.746082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2334.793134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2334.827070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2334.867678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2334.887781] LNet: 124028:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2334.903207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2334.967252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2335.039324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2335.109809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2335.161467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2335.290932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2336.379918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2336.636859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2336.678457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2336.709240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2336.762105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2336.811991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2336.856469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2336.881640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2336.917482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2336.933623] LNet: 124145:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2336.962488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2336.997500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2337.045771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2337.104725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2337.136755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2337.167579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2337.932033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.130@tcp [ 2338.342869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2338.424993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2338.492744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2338.542278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2338.610645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2338.684827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2338.726930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2338.764698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2338.802313] LNet: 124262:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2338.820844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2338.867731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2338.924297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2338.959842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2339.031318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2339.126642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2340.208815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2340.502486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2340.547468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2340.601337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2340.655654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2340.723533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2340.772410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2340.838235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2340.900290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2340.946555] LNet: 124380:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2340.997234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2341.097556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2341.167433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2341.240208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2341.315866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2341.381717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2347.547619] LNet: 124461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2347.551890] LNetError: 124461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2347.555608] LNet: Removed LNI 192.168.206.30@tcp [ 2347.569553] LNet: Removed LNI 192.168.206.30@tcp1 [ 2347.984887] Key type .llcrypt unregistered [ 2347.986450] Key type ._llcrypt unregistered [ 2362.876612] Key type ._llcrypt registered [ 2362.888698] Key type .llcrypt registered [ 2364.470411] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 20:10:46 (1784938246) [ 2366.449898] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2368.076475] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 20:10:50 (1784938250) [ 2374.330405] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 20:10:57 (1784938257) [ 2382.346301] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 20:11:05 (1784938265) [ 2390.016267] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 20:11:12 (1784938272) [ 2397.459323] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 20:11:20 (1784938280) [ 2405.015844] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 20:11:27 (1784938287) [ 2412.808269] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 20:11:35 (1784938295) [ 2420.518803] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 20:11:43 (1784938303) [ 2428.138629] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 20:11:50 (1784938310) [ 2436.828388] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 20:11:59 (1784938319) [ 2444.474616] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 20:12:07 (1784938327) [ 2452.462957] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 20:12:15 (1784938335) [ 2459.602566] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 20:12:21 (1784938341) [ 2467.212142] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 20:12:29 (1784938349) [ 2474.109477] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 20:12:37 (1784938357) [ 2480.893200] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 20:12:43 (1784938363) [ 2487.835475] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 20:12:50 (1784938370) [ 2494.104168] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 20:12:57 (1784938377) [ 2494.694824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2494.719199] alg: No test for adler32 (adler32-zlib) [ 2495.626679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2495.794880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2495.826147] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2496.914163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2496.927323] LNet: 135464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2496.933800] LNetError: 135464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2496.955604] LNet: Removed LNI 192.168.206.30@tcp [ 2496.986509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2497.046446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2497.089540] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2504.077384] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 20:13:06 (1784938386) [ 2510.814332] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 20:13:13 (1784938393) [ 2518.408717] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 20:13:21 (1784938401) [ 2525.280492] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 20:13:28 (1784938408) [ 2531.334801] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 20:13:34 (1784938414) [ 2537.925529] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 20:13:40 (1784938420) [ 2545.226925] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 20:13:47 (1784938427) [ 2553.466700] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 20:13:55 (1784938435) [ 2561.221293] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 20:14:04 (1784938444) [ 2567.839855] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 20:14:10 (1784938450) [ 2575.268431] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 20:14:18 (1784938458) [ 2583.067657] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 20:14:25 (1784938465) [ 2590.515258] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2352 sec ========== 20:14:33 (1784938473) [ 2590.801378] LNet: 142928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2590.818212] LNetError: 142928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2590.859626] LNet: Removed LNI 192.168.206.30@tcp [ 2591.578937] Key type .llcrypt unregistered [ 2591.582666] Key type ._llcrypt unregistered [ 2603.619200] Key type ._llcrypt registered [ 2603.628373] Key type .llcrypt registered [ 2603.917764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2603.966924] alg: No test for adler32 (adler32-zlib) [ 2605.202453] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 2605.661798] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2607.544283] Key type lgssc registered [ 2609.183269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2674.252712] Lustre: Mounted lustre-client [ 2679.211469] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2686.676118] Lustre: Unmounted lustre-client [ 2706.624716] Key type lgssc unregistered [ 2707.030045] LNet: 145855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2707.039771] LNetError: 145855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2707.072090] LNet: Removed LNI 192.168.206.30@tcp [ 2707.712160] Key type .llcrypt unregistered [ 2707.713973] Key type ._llcrypt unregistered