[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 450242657 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003238] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007931] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.011083] LSM: Security Framework initializing [ 0.012050] Yama: becoming mindful. [ 0.014006] SELinux: Initializing. [ 0.014966] *** VALIDATE selinux *** [ 0.022520] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027627] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030111] *** VALIDATE tmpfs *** [ 0.032337] *** VALIDATE proc *** [ 0.033220] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035263] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038026] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042426] debug: unmapping init [mem 0xffffffffa7059000-0xffffffffa7060fff] [ 0.044186] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045711] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046030] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050013] ... max period: 00007fffffffffff [ 0.051030] ... fixed-purpose events: 3 [ 0.052017] ... event mask: 000000070000000f [ 0.054239] rcu: Hierarchical SRCU implementation. [ 0.056513] smp: Bringing up secondary CPUs ... [ 0.057636] x86: Booting SMP configuration: [ 0.058029] .... node #0, CPUs: #1 #2 #3 [ 0.062184] smp: Brought up 1 node, 4 CPUs [ 0.064019] smpboot: Max logical packages: 1 [ 0.065023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.242305] node 0 deferred pages initialised in 171ms [ 0.247055] devtmpfs: initialized [ 0.248378] x86/mm: Memory block size: 128MB [ 0.252251] gcov: version magic: 0x41383552 [ 0.256454] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.261093] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.264372] pinctrl core: initialized pinctrl subsystem [ 0.266161] [ 0.266771] ************************************************************* [ 0.269018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.272016] ** ** [ 0.276034] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.278014] ** ** [ 0.282017] ** This means that this kernel is built to expose internal ** [ 0.284013] ** IOMMU data structures, which may compromise security on ** [ 0.287020] ** your system. ** [ 0.289013] ** ** [ 0.292016] ** If you see this message and you are not debugging the ** [ 0.294014] ** kernel, report this immediately to your vendor! ** [ 0.297014] ** ** [ 0.299015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.301015] ************************************************************* [ 0.304918] NET: Registered protocol family 16 [ 0.306457] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.309062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.312077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.316152] cpuidle: using governor menu [ 0.319045] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.322714] PCI: Using configuration type 1 for base access [ 0.325150] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.336149] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.339097] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.344046] cryptd: max_cpu_qlen set to 1000 [ 0.350455] ACPI: Added _OSI(Module Device) [ 0.352025] ACPI: Added _OSI(Processor Device) [ 0.354022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.356023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.361357] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.368620] ACPI: Interpreter enabled [ 0.370092] ACPI: PM: (supports S0 S3 S4 S5) [ 0.371044] ACPI: Using IOAPIC for interrupt routing [ 0.373152] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.377456] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.388370] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.390057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.393036] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.397059] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.403465] acpiphp: Slot [2] registered [ 0.405174] acpiphp: Slot [5] registered [ 0.407179] acpiphp: Slot [6] registered [ 0.408162] acpiphp: Slot [3] registered [ 0.410111] acpiphp: Slot [4] registered [ 0.412115] acpiphp: Slot [7] registered [ 0.414169] acpiphp: Slot [8] registered [ 0.415171] acpiphp: Slot [9] registered [ 0.417166] acpiphp: Slot [10] registered [ 0.419174] acpiphp: Slot [11] registered [ 0.421164] acpiphp: Slot [12] registered [ 0.423167] acpiphp: Slot [13] registered [ 0.424157] acpiphp: Slot [14] registered [ 0.426205] acpiphp: Slot [15] registered [ 0.428176] acpiphp: Slot [16] registered [ 0.430165] acpiphp: Slot [17] registered [ 0.431151] acpiphp: Slot [18] registered [ 0.433130] acpiphp: Slot [19] registered [ 0.435114] acpiphp: Slot [20] registered [ 0.436107] acpiphp: Slot [21] registered [ 0.438144] acpiphp: Slot [22] registered [ 0.440118] acpiphp: Slot [23] registered [ 0.442109] acpiphp: Slot [24] registered [ 0.443150] acpiphp: Slot [25] registered [ 0.445135] acpiphp: Slot [26] registered [ 0.446114] acpiphp: Slot [27] registered [ 0.448100] acpiphp: Slot [28] registered [ 0.449125] acpiphp: Slot [29] registered [ 0.451118] acpiphp: Slot [30] registered [ 0.453134] acpiphp: Slot [31] registered [ 0.455094] PCI host bridge to bus 0000:00 [ 0.456028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.459037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.461040] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.465042] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.468038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.471072] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.473243] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.477218] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.480538] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.490025] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.495931] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.499034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.502030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.505032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.508586] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.512161] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.515045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.517865] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.523014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.534925] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.540020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.546745] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.558027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.569031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.591018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.602948] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.612018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.629026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.641859] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.654191] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.657524] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.660465] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.663897] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.666342] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.671195] iommu: Default domain type: Passthrough [ 0.673652] SCSI subsystem initialized [ 0.675175] ACPI: bus type USB registered [ 0.677152] usbcore: registered new interface driver usbfs [ 0.679106] usbcore: registered new interface driver hub [ 0.681091] usbcore: registered new device driver usb [ 0.683221] pps_core: LinuxPPS API ver. 1 registered [ 0.685015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.689070] PTP clock support registered [ 0.691059] EDAC MC: Ver: 3.0.0 [ 0.693239] PCI: Using ACPI for IRQ routing [ 0.694601] NetLabel: Initializing [ 0.696016] NetLabel: domain hash size = 128 [ 0.697008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.699087] NetLabel: unlabeled traffic allowed by default [ 0.702005] vgaarb: loaded [ 0.704380] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.706028] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.713000] clocksource: Switched to clocksource kvm-clock [ 0.821249] VFS: Disk quotas dquot_6.6.0 [ 0.823504] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.826603] *** VALIDATE ramfs *** [ 0.828380] *** VALIDATE hugetlbfs *** [ 0.830833] pnp: PnP ACPI init [ 0.833599] pnp: PnP ACPI: found 6 devices [ 0.853910] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.858069] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.861185] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.863572] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.865754] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.868480] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.871412] NET: Registered protocol family 2 [ 0.873882] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.878571] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.883215] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.889119] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.894370] TCP: Hash tables configured (established 65536 bind 65536) [ 0.897792] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.901389] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.904450] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.907419] NET: Registered protocol family 1 [ 0.911603] RPC: Registered named UNIX socket transport module. [ 0.913389] RPC: Registered udp transport module. [ 0.914733] RPC: Registered tcp transport module. [ 0.915927] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.917811] NET: Registered protocol family 44 [ 0.919495] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.921535] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.923727] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.926455] PCI: CLS 0 bytes, default 64 [ 0.928225] Unpacking initramfs... [ 2.375919] debug: unmapping init [mem 0xffff9cf2fcc64000-0xffff9cf2fffcffff] [ 2.380405] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.382824] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.386212] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.965799] Initialise system trusted keyrings [ 2.967400] Key type blacklist registered [ 2.969028] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.980848] zbud: loaded [ 2.984984] *** VALIDATE nfs *** [ 2.986563] *** VALIDATE nfs4 *** [ 2.990158] pstore: using deflate compression [ 2.994975] Platform Keyring initialized [ 3.105438] NET: Registered protocol family 38 [ 3.107450] Key type asymmetric registered [ 3.109249] Asymmetric key parser 'x509' registered [ 3.111371] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.114796] io scheduler mq-deadline registered [ 3.116669] io scheduler kyber registered [ 3.118489] io scheduler bfq registered [ 3.120550] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.124061] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.127638] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.131090] ACPI: Power Button [PWRF] [ 3.138180] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.145229] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.160509] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.188852] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.217915] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.224883] Non-volatile memory driver v1.3 [ 3.226931] Linux agpgart interface v0.103 [ 3.262503] virtio_blk virtio1: [vda] 150608 512-byte logical blocks (77.1 MB/73.5 MiB) [ 3.268841] vda: detected capacity change from 0 to 77111296 [ 3.285603] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.290011] vdb: detected capacity change from 0 to 1073741824 [ 3.297275] libphy: Fixed MDIO Bus: probed [ 3.303242] usbcore: registered new interface driver usbserial_generic [ 3.307300] usbserial: USB Serial support registered for generic [ 3.310850] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.317415] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.320289] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.324408] mousedev: PS/2 mouse device common for all mice [ 3.328174] rtc_cmos 00:05: RTC can wake from S4 [ 3.331673] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.336993] rtc_cmos 00:05: registered as rtc0 [ 3.339559] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.340196] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.346074] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.346816] intel_pstate: CPU model not supported [ 3.353972] hid: raw HID events driver (C) Jiri Kosina [ 3.356559] usbcore: registered new interface driver usbhid [ 3.358951] usbhid: USB HID core driver [ 3.360788] drop_monitor: Initializing network drop monitor service [ 3.364042] Initializing XFRM netlink socket [ 3.365775] NET: Registered protocol family 10 [ 3.369146] Segment Routing with IPv6 [ 3.371345] NET: Registered protocol family 17 [ 3.373605] mpls_gso: MPLS GSO support [ 3.379292] RAS: Correctable Errors collector initialized. [ 3.381730] AVX version of gcm_enc/dec engaged. [ 3.383610] AES CTR mode by8 optimization enabled [ 3.466541] sched_clock: Marking stable (3466489766, 0)->(4375307701, -908817935) [ 3.469565] registered taskstats version 1 [ 3.471395] Loading compiled-in X.509 certificates [ 3.473131] zswap: loaded using pool lzo/zbud [ 3.499956] Key type big_key registered [ 3.513756] Key type encrypted registered [ 3.515406] ima: No TPM chip found, activating TPM-bypass! [ 3.517557] ima: Allocated hash algorithm: sha1 [ 3.519368] ima: No architecture policies found [ 3.520615] evm: Initialising EVM extended attributes: [ 3.522576] evm: security.selinux [ 3.523846] evm: security.ima [ 3.525126] evm: security.capability [ 3.526737] evm: HMAC attrs: 0x1 [ 3.529328] rtc_cmos 00:05: setting system clock to 2026-10-02 21:59:02 UTC (1790978342) [ 3.535318] debug: unmapping init [mem 0xffffffffa8003000-0xffffffffa81fffff] [ 3.538845] debug: unmapping init [mem 0xffffffffa6d82000-0xffffffffa7058fff] [ 3.551082] Write protecting the kernel read-only data: 28672k [ 3.555103] debug: unmapping init [mem 0xffffffffa5403000-0xffffffffa55fffff] [ 3.558131] debug: unmapping init [mem 0xffffffffa5d14000-0xffffffffa5dfffff] [ 3.589549] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.610630] systemd[1]: Detected virtualization kvm. [ 3.612319] systemd[1]: Detected architecture x86-64. [ 3.613955] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.640931] systemd[1]: No hostname configured. [ 3.643083] systemd[1]: Set hostname to . [ 3.645377] random: systemd: uninitialized urandom read (16 bytes read) [ 3.648297] systemd[1]: Initializing machine ID from random generator. [ 3.695106] random: ln: uninitialized urandom read (6 bytes read) [ 3.781253] random: systemd: uninitialized urandom read (16 bytes read) [ 3.784150] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.791288] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.794702] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.481543] device-mapper: uevent: version 1.0.3 [ 4.484466] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.238732] virtio_net virtio0 ens2: renamed from eth0 [ 5.332429] scsi host0: ata_piix [ 5.338669] scsi host1: ata_piix [ 5.340292] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.343392] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.873750] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.032527] random: crng init done [ 10.034905] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.454566] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.652893] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.951041] SELinux: Disabled at runtime. [ 12.014978] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.032047] systemd[1]: Detected virtualization kvm. [ 12.035857] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.643749] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.649543] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.657813] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.662747] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.666751] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.675982] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.684345] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Soc[ 12.767047] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.297671] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.566596] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.569341] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.734726] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.745783] EDAC sbridge: Ver: 1.1.2 [ 14.829915] Key type dns_resolver registered [ 15.154548] NFS: Registering the id_resolver key type [ 15.158347] Key type id_resolver registered [ 15.161664] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg422-client login: [ 44.793971] libcfs: loading out-of-tree module taints kernel. [ 44.870762] Key type ._llcrypt registered [ 44.872752] Key type .llcrypt registered [ 45.113166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 45.122174] alg: No test for adler32 (adler32-zlib) [ 46.238282] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 46.641725] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 48.304049] Key type lgssc registered [ 51.162647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 56.392037] hrtimer: interrupt took 4384915 ns [ 212.635499] Lustre: Mounted lustre-client - version 2.17.59_2_ge06ed12 [ 218.442521] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 232.245948] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing check_logdir /tmp/testlogs/ [ 238.176670] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing yml_node [ 238.560185] Lustre: lustre-OST0000-osc-ffff9cf368b7e000: disconnect after 23s idle [ 243.696731] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 247.031784] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 250.357198] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 252.382460] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Oct 2 18:03:09 EDT 2026 [ 276.201581] Lustre: Unmounted lustre-client [ 312.823669] Key type lgssc unregistered [ 313.198257] LNet: 8413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.209608] LNetError: 8413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 313.275390] LNet: Removed LNI 192.168.204.22@tcp [ 314.552685] Key type .llcrypt unregistered [ 314.557821] Key type ._llcrypt unregistered [ 348.992338] Key type ._llcrypt registered [ 348.994726] Key type .llcrypt registered [ 349.303419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 349.327528] alg: No test for adler32 (adler32-zlib) [ 350.294444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 350.403783] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 350.426684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 355.365973] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 355.482824] LNet: 9300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.490439] LNetError: 9300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 355.527387] LNet: Removed LNI 192.168.204.22@tcp [ 356.458827] Key type .llcrypt unregistered [ 356.462661] Key type ._llcrypt unregistered [ 363.807821] Key type ._llcrypt registered [ 363.810850] Key type .llcrypt registered [ 365.099717] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 18:05:02 (1790978702) [ 365.467225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 365.476784] alg: No test for adler32 (adler32-zlib) [ 366.408044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.504548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.526479] LNet: 9721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.561385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.616458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 366.641680] LNet: 9728:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 374.909963] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 18:05:12 (1790978712) [ 375.224920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.280139] LNet: 10308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.306547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.410237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 384.247947] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 18:05:21 (1790978721) [ 384.413523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.471598] LNet: 10887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.518826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.655779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 394.425102] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 18:05:31 (1790978731) [ 394.590299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.611815] LNet: 11472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.654980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.805779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 403.186507] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 18:05:40 (1790978740) [ 403.282033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.309782] LNet: 12057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.360252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.486140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 404.125129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 413.595166] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 18:05:50 (1790978750) [ 413.755742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.795675] LNet: 12648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.851324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.976926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 414.949250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 415.674276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 424.094815] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 18:06:01 (1790978761) [ 424.247377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.359764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.489295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 433.570248] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 18:06:10 (1790978770) [ 433.923212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.944575] LNet: 13837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 433.949138] LNet: 13837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 433.986770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.107894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 434.204765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 434.684333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 434.707962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 434.988350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 435.088566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 435.664224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 435.716964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 436.304449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 436.366787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 436.971958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 437.017150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 437.504885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 437.545554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 438.183615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 438.289312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 439.218320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 439.309415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 440.143403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 440.211421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 441.086381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 441.143407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 441.771638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 441.802197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 442.102619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 442.172406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 453.819754] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 18:06:30 (1790978790) [ 453.998904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.061919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.198408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 454.232152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 462.733548] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 18:06:40 (1790978800) [ 463.070743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.136519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.251890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 463.314609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 471.876307] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 18:06:49 (1790978809) [ 472.006994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.064471] LNet: 15687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 472.080382] LNet: 15687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 472.125053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.299794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 472.345317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 480.197450] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 18:06:57 (1790978817) [ 480.579373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.662193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.795529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 480.853398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 488.367880] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 18:07:06 (1790978826) [ 488.468581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.556187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.659708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 488.696921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 495.762126] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 18:07:13 (1790978833) [ 495.971748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.034386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.124567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 496.192136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 503.177401] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 18:07:20 (1790978840) [ 503.494136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.551420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.695290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 504.313295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 510.200705] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 18:07:28 (1790978848) [ 510.396723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.423266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.632266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.861474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 517.027856] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 18:07:34 (1790978854) [ 517.281271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.316519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.472640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 517.643701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 524.790668] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 18:07:42 (1790978862) [ 525.071942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.108193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.793395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 526.176304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 533.486099] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 18:07:51 (1790978871) [ 533.827829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.881700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.961582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 534.042410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 541.390115] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 18:07:58 (1790978878) [ 541.623347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.650355] LNet: 21114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.658517] LNet: 21114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 541.675830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.741400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 541.824944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 550.419828] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 18:08:07 (1790978887) [ 550.541401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.621754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.761496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 559.159814] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 18:08:16 (1790978896) [ 559.259682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.305307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.393195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 567.731104] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 18:08:24 (1790978904) [ 567.981433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.100601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.377841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 569.071881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 581.433702] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 18:08:37 (1790978917) [ 582.113950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.189813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.366619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 582.441948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 593.729631] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 18:08:50 (1790978930) [ 593.944630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.012621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.181623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 594.236518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 604.084705] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 18:09:01 (1790978941) [ 604.598839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.648230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.810946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 604.887168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 613.565276] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 18:09:10 (1790978950) [ 613.971835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.001444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.107768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 614.174086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 622.496205] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 18:09:19 (1790978959) [ 622.641825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.682898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.744877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 622.817277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 622.838235] LNetError: 25841:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 622.872241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 622.894859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 630.553499] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 18:09:27 (1790978967) [ 630.653402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.749325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.914356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.y5p9jm1qTR [ 631.032163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.y5p9jm1qTR [ 631.137950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.y5p9jm1qTR [ 637.675511] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 18:09:35 (1790978975) [ 637.844837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.947955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.057232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 638.090595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 638.158696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 647.962346] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 18:09:44 (1790978984) [ 648.188089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.285826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.493716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 648.548850] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 650.328569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 652.219753] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 654.399614] Key type lgssc registered [ 656.316825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 661.164990] LNetError: 27603:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.22@tcp99 has no available nets [ 664.366974] Key type lgssc unregistered [ 664.791299] LNetError: 28719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.822139] LNet: Removed LNI 192.168.204.22@tcp [ 665.481588] Key type .llcrypt unregistered [ 665.484205] Key type ._llcrypt unregistered [ 673.039619] Key type ._llcrypt registered [ 673.041959] Key type .llcrypt registered [ 674.441577] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 18:10:11 (1790979011) [ 675.178404] Key type .llcrypt unregistered [ 675.182062] Key type ._llcrypt unregistered [ 682.058164] Key type ._llcrypt registered [ 682.061303] Key type .llcrypt registered [ 682.212960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 682.230800] alg: No test for adler32 (adler32-zlib) [ 683.215946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.319622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 683.355569] LNet: 29642:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 683.361268] LNet: 29642:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 683.375278] LNet: 29642:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 683.399794] LNet: Message forwarding will be enabled in 2s [ 686.495114] LNet: Message forwarding enabled [ 687.146739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 687.177984] LNet: Message forwarding will stop in 2s to 6s [ 690.600850] LNet: Message forwarding disabled [ 690.790636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 690.826485] LNet: 29731:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 690.840982] LNet: 29731:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 690.853771] LNet: 29731:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 690.876918] LNet: Message forwarding will be enabled in 2s [ 691.065619] LNet: 29744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.976782] Key type .llcrypt unregistered [ 691.981410] Key type ._llcrypt unregistered [ 708.103028] Key type ._llcrypt registered [ 708.108960] Key type .llcrypt registered [ 709.235257] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 18:10:46 (1790979046) [ 710.327166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 710.336942] alg: No test for adler32 (adler32-zlib) [ 711.275693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.379696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 718.517534] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 18:10:55 (1790979055) [ 718.689281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.779925] LNet: 31141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.836269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.969331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 719.030487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 719.090201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 719.148484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 719.212526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 719.294855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 719.357210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 719.418822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 719.482242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 719.588848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 719.673693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 719.727536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 719.809994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 719.894817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 719.961973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 720.042470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 720.104058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 720.162361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 720.233357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 720.288819] LNetError: 31191:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 727.087352] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 18:11:04 (1790979064) [ 727.272181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.291907] LNet: 31762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.314754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.385699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 734.518028] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 18:11:12 (1790979072) [ 736.878423] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg422-server [ 738.846503] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 18:11:16 (1790979076) [ 739.248443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 739.287575] LNet: 32694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.326776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.409615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 739.444587] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 740.526204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[23-27]@tcp [ 740.556301] LNet: 32827:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 740.625943] LNetError: 32698:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.24@tcp is being used as a gateway but routing feature is not turned on [ 740.645732] LNetError: 32698:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.24@tcp has gone from up to down [ 748.934987] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 18:11:26 (1790979086) [ 751.188733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg422-server [ 752.988386] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 18:11:30 (1790979090) [ 753.541853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.569811] LNet: 33763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.576668] LNetError: 33763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.607334] LNet: Removed LNI 192.168.204.22@tcp [ 753.700926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.826039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 753.862475] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 754.882213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[23-28/2]@tcp [ 754.915092] LNet: 33880:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 754.926404] LNet: 33880:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 754.969195] LNetError: 33767:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.23@tcp is being used as a gateway but routing feature is not turned on [ 754.974147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[23-28/2]@tcp [ 754.990628] LNetError: 33767:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 754.990670] LNetError: 33767:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.23@tcp has gone from up to down [ 754.990674] LNetError: 33767:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 762.990620] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 18:11:40 (1790979100) [ 763.175344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.211601] LNet: 34468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.223771] LNetError: 34468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 764.262360] LNet: Removed LNI 192.168.204.22@tcp [ 764.316214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.482932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 764.590424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 764.660512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.746541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 764.799658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 764.832326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.877441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 764.945516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 764.984451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.059823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 765.154707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.203445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.246040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 765.316542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.350972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.424714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 765.535370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.600520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.711891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 765.823960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.901760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.023421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 766.131057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.194451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.268883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 766.375210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.413931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.477985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 766.553903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.586364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.636764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 766.692271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.728797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.807552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 773.856975] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 18:11:51 (1790979111) [ 776.277433] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg422-server [ 778.345515] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 18:11:55 (1790979115) [ 781.466596] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg422-server [ 784.012498] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 18:12:00 (1790979120) [ 784.127362] LNet: 35868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.134782] LNet: 35868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 784.934955] Key type .llcrypt unregistered [ 784.939439] Key type ._llcrypt unregistered [ 792.397730] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 792.634105] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 792.646415] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 792.652837] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 795.445366] Key type ._llcrypt registered [ 795.450784] Key type .llcrypt registered [ 795.727129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 795.772064] alg: No test for adler32 (adler32-zlib) [ 796.746924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.831925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 796.871363] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 797.554498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 797.575359] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 799.523903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 799.545979] LNet: Removed LNI 10.1.2.3@tcp [ 799.698545] LNet: 36594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.713549] LNetError: 36594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.724905] LNet: Removed LNI 192.168.204.22@tcp [ 800.390530] Key type .llcrypt unregistered [ 800.392402] Key type ._llcrypt unregistered [ 812.898178] Key type ._llcrypt registered [ 812.905942] Key type .llcrypt registered [ 814.305777] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 18:12:31 (1790979151) [ 814.901918] Key type .llcrypt unregistered [ 814.905630] Key type ._llcrypt unregistered [ 820.900381] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 821.120617] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 821.135415] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 821.145382] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 823.807240] Key type ._llcrypt registered [ 823.810205] Key type .llcrypt registered [ 824.075453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 824.116691] alg: No test for adler32 (adler32-zlib) [ 825.114445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.178860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 825.199386] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 825.926655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 825.954444] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 828.763538] LNet: 38092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.775526] LNetError: 38092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 828.782561] LNet: Removed LNI 10.1.2.3@tcp [ 828.813074] LNet: Removed LNI 192.168.204.22@tcp [ 829.618566] Key type .llcrypt unregistered [ 829.622314] Key type ._llcrypt unregistered [ 841.913664] Key type ._llcrypt registered [ 841.915423] Key type .llcrypt registered [ 843.405733] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 18:13:00 (1790979180) [ 844.281630] Key type .llcrypt unregistered [ 844.285510] Key type ._llcrypt unregistered [ 851.711709] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 852.011451] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 852.027314] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 852.035425] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 854.743528] Key type ._llcrypt registered [ 854.745259] Key type .llcrypt registered [ 855.054665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 855.090786] alg: No test for adler32 (adler32-zlib) [ 856.033461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.080716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 856.112306] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 858.380159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 858.405819] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 860.379825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 860.403244] LNet: Removed LNI 10.1.2.3@tcp [ 860.420609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 860.449970] LNet: Removed LNI 10.1.2.31@tcp [ 860.466309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 860.499461] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 861.414975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 861.436083] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 862.394912] LNet: 40009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.402053] LNetError: 40009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 862.413492] LNet: Removed LNI 10.1.2.31@tcp [ 862.434134] LNet: Removed LNI 10.1.2.3@tcp [ 863.209646] Key type .llcrypt unregistered [ 863.215162] Key type ._llcrypt unregistered [ 876.558858] Key type ._llcrypt registered [ 876.566586] Key type .llcrypt registered [ 878.176788] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 18:13:35 (1790979215) [ 878.683358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 878.713716] alg: No test for adler32 (adler32-zlib) [ 879.661264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.730594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 879.748419] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 880.433954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.463390] LNet: 40961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.475912] LNetError: 40961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.503900] LNet: Removed LNI 192.168.204.22@tcp [ 880.524291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.605460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.22@tcp [ 880.637902] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 881.371993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.22@tcp [ 881.405530] LNet: Removed LNI 192.168.204.22@tcp [ 881.448333] LNet: 41097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.072361] Key type .llcrypt unregistered [ 882.075803] Key type ._llcrypt unregistered [ 893.768170] Key type ._llcrypt registered [ 893.770201] Key type .llcrypt registered [ 895.177839] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 18:13:52 (1790979232) [ 897.157287] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg422-server [ 899.015624] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 18:13:56 (1790979236) [ 899.512305] Key type .llcrypt unregistered [ 899.516764] Key type ._llcrypt unregistered [ 905.508856] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 905.788305] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 905.819285] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 905.832816] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 908.386632] Key type ._llcrypt registered [ 908.389260] Key type .llcrypt registered [ 908.609828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 908.642563] alg: No test for adler32 (adler32-zlib) [ 909.673898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.787826] LNet: Added LNI 192.168.204.22@tcp1 [8/256/0/180] [ 910.793890] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 913.444859] LNet: 42935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.457669] LNetError: 42935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.468922] LNet: Removed LNI 192.168.204.22@tcp1 [ 913.480883] LNet: Removed LNI 10.1.2.3@tcp2 [ 914.356426] Key type .llcrypt unregistered [ 914.361714] Key type ._llcrypt unregistered [ 927.260651] Key type ._llcrypt registered [ 927.264865] Key type .llcrypt registered [ 928.731911] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 18:14:26 (1790979266) [ 929.066501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 929.076598] alg: No test for adler32 (adler32-zlib) [ 930.088014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.137549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 930.156694] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 930.928601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.950569] LNet: 43904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.957567] LNetError: 43904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.979863] LNet: Removed LNI 192.168.204.22@tcp [ 931.000212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.072307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 931.099328] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 938.985632] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 18:14:36 (1790979276) [ 939.177789] LNet: 44487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.187474] LNetError: 44487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.207170] LNet: Removed LNI 192.168.204.22@tcp [ 939.936570] Key type .llcrypt unregistered [ 939.939185] Key type ._llcrypt unregistered [ 946.572245] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 946.808152] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 946.813875] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 946.821554] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 949.404970] Key type ._llcrypt registered [ 949.414145] Key type .llcrypt registered [ 949.594173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 949.612530] alg: No test for adler32 (adler32-zlib) [ 950.562967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.607300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 950.631248] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 951.341510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 951.361988] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 953.646667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.662765] LNet: 45236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.666528] LNetError: 45236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 953.671921] LNet: Removed LNI 10.1.2.3@tcp [ 953.681046] LNet: Removed LNI 192.168.204.22@tcp [ 953.699408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.759143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 953.799233] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 953.812285] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 954.454585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.477665] LNet: 45263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.483791] LNetError: 45263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 954.493422] LNetError: 45263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 954.501445] LNet: Removed LNI 10.1.2.3@tcp [ 954.520577] LNet: Removed LNI 192.168.204.22@tcp [ 954.553927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.626697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 954.659794] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 954.663612] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 955.269596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 955.302933] LNet: Added LNI 192.168.204.22@tcp2 [8/256/0/180] [ 956.600756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.642865] LNet: 45425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.663945] LNetError: 45425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 956.679751] LNetError: 45425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 956.690286] LNet: Removed LNI 10.1.2.3@tcp [ 956.694830] LNet: Removed LNI 192.168.204.22@tcp [ 956.715090] LNet: Removed LNI 192.168.204.22@tcp2 [ 956.737609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.793507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 956.821798] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 956.830244] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 956.835932] LNet: Added LNI 192.168.204.22@tcp2 [8/256/0/180] [ 965.692816] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 18:15:02 (1790979302) [ 965.810914] LNet: 46015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.820290] LNetError: 46015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 965.831110] LNetError: 46015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 965.845777] LNet: Removed LNI 10.1.2.3@tcp [ 965.852051] LNet: Removed LNI 192.168.204.22@tcp [ 965.866973] LNet: Removed LNI 192.168.204.22@tcp2 [ 966.625675] Key type .llcrypt unregistered [ 966.631056] Key type ._llcrypt unregistered [ 973.488664] Key type ._llcrypt registered [ 973.491555] Key type .llcrypt registered [ 973.861736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 973.895295] alg: No test for adler32 (adler32-zlib) [ 974.874966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.974574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 975.030862] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 976.217233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 976.255894] LNet: 46417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.271843] LNetError: 46417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 976.292702] LNet: Removed LNI 192.168.204.22@tcp [ 976.316796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 976.395979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 976.421511] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 977.002799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 977.044729] LNet: 46447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.073192] LNetError: 46447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.099088] LNet: Removed LNI 192.168.204.22@tcp [ 977.146948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.272598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 977.335666] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 978.200299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.231130] LNet: 46477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.234878] LNetError: 46477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.249019] LNet: Removed LNI 192.168.204.22@tcp [ 978.265902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.326755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 978.346413] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 978.893339] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 979.025962] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 979.027852] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 979.030224] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 981.405886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.433918] LNet: 46558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.443033] LNetError: 46558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.465804] LNet: Removed LNI 192.168.204.22@tcp [ 981.498260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.573142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 981.606395] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 989.176393] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 18:15:26 (1790979326) [ 989.346763] LNet: 47155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.355207] LNetError: 47155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.384664] LNet: Removed LNI 192.168.204.22@tcp [ 990.105149] Key type .llcrypt unregistered [ 990.109227] Key type ._llcrypt unregistered [ 995.797789] Key type ._llcrypt registered [ 995.800769] Key type .llcrypt registered [ 996.081693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 996.103578] alg: No test for adler32 (adler32-zlib) [ 997.016476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.068460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 997.103897] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 997.920492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.936852] LNet: 47554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.944940] LNetError: 47554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.968177] LNet: Removed LNI 192.168.204.22@tcp [ 997.989206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.047677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 998.087967] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1006.318568] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 18:15:43 (1790979343) [ 1006.526246] LNet: 48140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.553082] LNetError: 48140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.590221] LNet: Removed LNI 192.168.204.22@tcp [ 1007.516490] Key type .llcrypt unregistered [ 1007.518342] Key type ._llcrypt unregistered [ 1012.905202] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1013.139684] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1013.158038] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1013.169922] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1015.829114] Key type ._llcrypt registered [ 1015.833877] Key type .llcrypt registered [ 1016.160386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1016.193828] alg: No test for adler32 (adler32-zlib) [ 1017.174909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1017.307989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1017.346586] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1018.743454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1018.765743] LNet: 48597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.779823] LNetError: 48597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.798528] LNet: Removed LNI 192.168.204.22@tcp [ 1018.851563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.953695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1019.019898] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1026.130414] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 18:16:03 (1790979363) [ 1026.262957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1026.290330] LNet: 49195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.305036] LNetError: 49195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.332354] LNet: Removed LNI 192.168.204.22@tcp [ 1026.359632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1026.452771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1032.048838] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 18:16:09 (1790979369) [ 1032.180267] LNet: 49774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.735257] Key type .llcrypt unregistered [ 1032.740918] Key type ._llcrypt unregistered [ 1038.709274] Key type ._llcrypt registered [ 1038.715572] Key type .llcrypt registered [ 1038.888231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1038.906910] alg: No test for adler32 (adler32-zlib) [ 1039.928989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1040.052795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1040.092994] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1041.101420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1041.131311] LNet: 50175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.137803] LNetError: 50175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.149707] LNet: Removed LNI 192.168.204.22@tcp [ 1041.168981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.223616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1041.251257] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1047.602623] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 18:16:25 (1790979385) [ 1047.754479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.784717] LNet: 50766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.801707] LNetError: 50766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.832633] LNet: Removed LNI 192.168.204.22@tcp [ 1047.847653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.909642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1053.914503] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 18:16:31 (1790979391) [ 1055.348770] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1057.154779] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 18:16:34 (1790979394) [ 1058.663990] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1060.711890] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 18:16:37 (1790979397) [ 1062.529554] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1064.391745] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 18:16:41 (1790979401) [ 1066.004201] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1067.897481] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 18:16:45 (1790979405) [ 1069.701055] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1071.407769] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 18:16:49 (1790979409) [ 1071.505145] LNet: 52784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.406817] Key type .llcrypt unregistered [ 1072.412052] Key type ._llcrypt unregistered [ 1079.128081] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1079.440805] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1079.450944] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1079.470643] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1082.011971] Key type ._llcrypt registered [ 1082.013611] Key type .llcrypt registered [ 1082.238787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1082.248778] alg: No test for adler32 (adler32-zlib) [ 1083.145562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.263210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1083.290656] LNet: Added LNI 192.168.204.22@tcp [8/512/0/180] [ 1083.296304] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1083.300586] LNet: 53109:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1083.406808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1083.430758] LNet: 53128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.437609] LNetError: 53128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1083.450505] LNet: Removed LNI 10.1.2.3@tcp [ 1083.454582] LNetError: 53128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.469126] LNet: Removed LNI 192.168.204.22@tcp [ 1083.509242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.590082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1083.621636] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1083.689408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1083.703974] LNet: 53152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.708692] LNetError: 53152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.744881] LNet: Removed LNI 192.168.204.22@tcp [ 1083.763262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.820954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1083.869901] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1083.875050] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1083.882402] LNet: 53160:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1083.973442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1083.988697] LNet: 53178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.993796] LNetError: 53178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1084.000598] LNet: Removed LNI 10.1.2.3@tcp [ 1084.002936] LNetError: 53178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.017192] LNet: Removed LNI 192.168.204.22@tcp [ 1084.058570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1084.115696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1084.132551] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1084.136953] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1084.139874] LNet: 53186:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1084.347298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1084.371481] LNet: 53212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.384907] LNetError: 53212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1084.390777] LNet: Removed LNI 10.1.2.3@tcp [ 1084.397539] LNetError: 53212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.425326] LNet: Removed LNI 192.168.204.22@tcp [ 1084.463923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1084.528151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1084.558611] LNet: Added LNI 192.168.204.22@tcp [8/256/0/182] [ 1084.567880] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1084.666840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1084.686700] LNet: 53241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.700703] LNetError: 53241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1084.707324] LNet: Removed LNI 10.1.2.3@tcp [ 1084.710175] LNetError: 53241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.724529] LNet: Removed LNI 192.168.204.22@tcp [ 1084.769506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1084.865330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1084.903601] LNet: Added LNI 192.168.204.22@tcp [8/256/0/182] [ 1084.911077] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1084.913311] LNet: 53249:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1084.979234] LNet: Removed LNI 10.1.2.3@tcp [ 1084.991853] LNet: Removed LNI 192.168.204.22@tcp [ 1085.597897] Key type .llcrypt unregistered [ 1085.599450] Key type ._llcrypt unregistered [ 1096.866779] Key type ._llcrypt registered [ 1096.869519] Key type .llcrypt registered [ 1098.076660] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 18:17:15 (1790979435) [ 1099.755214] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1101.950084] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 18:17:19 (1790979439) [ 1102.386870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1102.405170] alg: No test for adler32 (adler32-zlib) [ 1103.451548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1103.531847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1103.575337] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1104.246631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.22 [ 1104.264634] LNet: 54512:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1104.277835] LNet: Removed LNI 192.168.204.22@tcp [ 1110.221826] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 18:17:27 (1790979447) [ 1110.381111] LNet: 55079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.385972] LNetError: 55079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1110.396393] LNet: Removed LNI 192.168.204.22@tcp [ 1111.019453] Key type .llcrypt unregistered [ 1111.022338] Key type ._llcrypt unregistered [ 1117.324673] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1117.582324] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1120.401497] Key type ._llcrypt registered [ 1120.405828] Key type .llcrypt registered [ 1120.649907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1120.678994] alg: No test for adler32 (adler32-zlib) [ 1122.012517] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1128.937588] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 18:17:46 (1790979466) [ 1129.124666] LNet: 55996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.129767] LNetError: 55996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1129.142567] LNet: Removed LNI 10.1.2.3@tcp [ 1129.830369] Key type .llcrypt unregistered [ 1129.832103] Key type ._llcrypt unregistered [ 1137.423875] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1137.835270] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1137.838631] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1137.853479] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1140.730679] Key type ._llcrypt registered [ 1140.733873] Key type .llcrypt registered [ 1140.918952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1140.935424] alg: No test for adler32 (adler32-zlib) [ 1141.979834] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1149.963557] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 18:18:07 (1790979487) [ 1151.713615] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1154.013085] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 18:18:11 (1790979491) [ 1154.193599] LNet: 57177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.203199] LNetError: 57177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1154.248091] LNet: Removed LNI 10.1.2.3@tcp [ 1154.806585] Key type .llcrypt unregistered [ 1154.809340] Key type ._llcrypt unregistered [ 1161.659510] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1162.147691] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1162.154601] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1162.167496] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1163.792097] Key type ._llcrypt registered [ 1163.794605] Key type .llcrypt registered [ 1164.013310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1164.028398] alg: No test for adler32 (adler32-zlib) [ 1164.941479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1165.000988] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1173.631146] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 18:18:31 (1790979511) [ 1174.031930] LNet: 58402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1174.043087] LNetError: 58402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1174.061661] LNet: Removed LNI 10.1.2.3@tcp [ 1174.656448] Key type .llcrypt unregistered [ 1174.659850] Key type ._llcrypt unregistered [ 1180.744756] Key type ._llcrypt registered [ 1180.752977] Key type .llcrypt registered [ 1180.985721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1180.997777] alg: No test for adler32 (adler32-zlib) [ 1181.932302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1181.972151] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1191.461349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 1192.384653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_interrupt -r 1 [ 1192.448251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_interrupt -r 1 [ 1192.522137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1192.561286] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1192.987142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1193.023635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1193.059521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1193.104294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1193.162170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1193.216175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1193.247552] LNet: 1 local NIs in recovery (showing 1): 192.168.204.22@tcp [ 1193.251276] LNet: There was an unexpected network error while writing to 192.168.204.22: rc = -22 [ 1193.280876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1193.329690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1193.372736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1193.385684] LNet: 58841:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1193.423306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1193.466706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1193.502484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1193.564503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1193.636304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.700687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1193.768903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1194.447385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_dropped -r 1 [ 1194.504857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_dropped -r 1 [ 1194.547544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1194.578219] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1194.914662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1194.968378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1195.002971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1195.026648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1195.050217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1195.077884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1195.114345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1195.146785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1195.176408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1195.198090] LNet: 58953:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1195.222238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1195.278566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1195.325376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1195.372515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1195.433723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1195.490681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1195.557103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1196.043161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_aborted -r 1 [ 1196.091275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_aborted -r 1 [ 1196.146828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1196.448260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1196.478936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1196.524295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1196.567207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1196.623187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1196.663839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1196.702847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1196.735962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1196.762477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1196.787283] LNet: 59064:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1196.805830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1196.832533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1196.863708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1196.885479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1196.912934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1196.938229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1196.966201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1197.473703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_no_route -r 1 [ 1197.504746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_no_route -r 1 [ 1197.542778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1197.571317] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1197.577618] LNet: Skipped 2 previous similar messages [ 1197.856162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1197.927134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1197.990804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1198.051923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1198.095615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1198.143667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1198.189342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1198.226448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1198.267755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1198.307362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1198.340705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1198.381730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1198.409049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1198.443666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1198.480080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1198.532319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1198.998923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_timeout -r 1 [ 1199.035330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_timeout -r 1 [ 1199.074973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1199.314379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1199.341545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1199.381493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1199.403724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1199.426253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1199.459175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1199.487696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1199.523470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1199.565540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1199.583712] LNet: 59286:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1199.596462] LNet: 59286:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1199.634306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1199.684710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1199.721182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1199.748575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1199.779787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1199.817632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1199.846312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1200.342885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_error -r 1 [ 1200.371903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_error -r 1 [ 1200.402778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1200.616383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1200.645323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1200.671974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1200.695372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1200.717994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1200.748917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1200.780943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1200.809969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1200.842538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1200.876689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1200.921381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1200.961349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1201.000156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1201.033521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1201.060636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1201.111651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1201.469648] LNet: 59439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1201.478984] LNetError: 59439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1201.497122] LNet: Removed LNI 192.168.204.22@tcp [ 1201.905571] Key type .llcrypt unregistered [ 1201.907687] Key type ._llcrypt unregistered [ 1221.086696] Key type ._llcrypt registered [ 1221.088762] Key type .llcrypt registered [ 1222.798815] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 18:19:19 (1790979559) [ 1223.586851] Key type .llcrypt unregistered [ 1223.590727] Key type ._llcrypt unregistered [ 1229.777796] Key type ._llcrypt registered [ 1229.780196] Key type .llcrypt registered [ 1230.039920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1230.064515] alg: No test for adler32 (adler32-zlib) [ 1231.067680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1231.127490] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1238.776479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 1250.364422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1250.406702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1250.441465] LNet: Added LNI 192.168.204.22@tcp1 [8/256/0/180] [ 1251.301662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 1251.940118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_interrupt -r 1 [ 1251.980758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp1 -m GET -e local_interrupt -r 1 [ 1252.041031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_interrupt -r 1 [ 1252.080650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -m GET -e local_interrupt -r 1 [ 1252.137704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp -m GET -e local_interrupt -r 1 [ 1252.179290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp1 -m GET -e local_interrupt -r 1 [ 1252.228705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -m GET -e local_interrupt -r 1 [ 1252.284414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -m GET -e local_interrupt -r 1 [ 1252.343176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1252.384888] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1252.399624] LNet: 1 local NIs in recovery (showing 1): 192.168.204.22@tcp1 [ 1252.679759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1252.712178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1252.736203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1252.776568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1252.817882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1252.842203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1252.863382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1252.887439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1252.909753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1252.926498] LNet: 60915:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1252.940728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1252.966414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1252.993387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1253.024990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1253.080884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1253.158875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1253.211919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1253.799982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_dropped -r 1 [ 1253.833377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp1 -m GET -e local_dropped -r 1 [ 1253.868744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_dropped -r 1 [ 1253.897185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -m GET -e local_dropped -r 1 [ 1253.932541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp -m GET -e local_dropped -r 1 [ 1253.973047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp1 -m GET -e local_dropped -r 1 [ 1253.997819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -m GET -e local_dropped -r 1 [ 1254.057679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -m GET -e local_dropped -r 1 [ 1254.095834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1254.117869] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1254.122931] LNet: Skipped 2 previous similar messages [ 1254.334407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1254.365700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1254.417330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1254.459360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1254.499159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1254.539080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1254.587574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1254.631842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1254.662187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1254.688149] LNet: 61038:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1254.712751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1254.746881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1254.773951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1254.804360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1254.832443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1254.869127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1254.898610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1255.410338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_aborted -r 1 [ 1255.503785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp1 -m GET -e local_aborted -r 1 [ 1255.568384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_aborted -r 1 [ 1255.630893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -m GET -e local_aborted -r 1 [ 1255.699611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp -m GET -e local_aborted -r 1 [ 1255.768888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp1 -m GET -e local_aborted -r 1 [ 1255.850785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -m GET -e local_aborted -r 1 [ 1255.964491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -m GET -e local_aborted -r 1 [ 1256.029573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1256.047607] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1256.052228] LNet: Skipped 2 previous similar messages [ 1256.404823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1256.441821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1256.464113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1256.496696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1256.530169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1256.559410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1256.598626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1256.650431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1256.725458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1256.768840] LNet: 61162:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1256.797144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1256.841189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1256.878838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1256.905573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1256.931954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1256.966238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1256.997106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1257.507989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_no_route -r 1 [ 1257.538795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp1 -m GET -e local_no_route -r 1 [ 1257.572463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_no_route -r 1 [ 1257.611478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -m GET -e local_no_route -r 1 [ 1257.646185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp -m GET -e local_no_route -r 1 [ 1257.678436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp1 -m GET -e local_no_route -r 1 [ 1257.713837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -m GET -e local_no_route -r 1 [ 1257.740373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -m GET -e local_no_route -r 1 [ 1257.772977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1258.029820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1258.056386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1258.079885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1258.104322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1258.133184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1258.156463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1258.209911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1258.254213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1258.280341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1258.309281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1258.331935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1258.356157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1258.377569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1258.412171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1258.465844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1258.495962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1258.959747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_timeout -r 1 [ 1258.992186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp1 -m GET -e local_timeout -r 1 [ 1259.025861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_timeout -r 1 [ 1259.054521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -m GET -e local_timeout -r 1 [ 1259.083637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp -m GET -e local_timeout -r 1 [ 1259.115228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp1 -m GET -e local_timeout -r 1 [ 1259.141878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -m GET -e local_timeout -r 1 [ 1259.170483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -m GET -e local_timeout -r 1 [ 1259.214622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1259.239380] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1259.245214] LNet: Skipped 5 previous similar messages [ 1259.424684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1259.459601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1259.493332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1259.530834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1259.575141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1259.606901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1259.660230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1259.697252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1259.728413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1259.753507] LNet: 61408:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1259.762365] LNet: 61408:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1259.782723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1259.822369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1259.853386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1259.894407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1259.922667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1259.962864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1260.021975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1260.648862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_error -r 1 [ 1260.680820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp1 -m GET -e local_error -r 1 [ 1260.714765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_error -r 1 [ 1260.746513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -m GET -e local_error -r 1 [ 1260.777970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp -m GET -e local_error -r 1 [ 1260.808249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp1 -m GET -e local_error -r 1 [ 1260.838465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -m GET -e local_error -r 1 [ 1260.873176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -m GET -e local_error -r 1 [ 1260.907288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1261.149489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1261.204399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1261.256466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1261.299755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1261.357338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1261.407696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1261.448893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1261.486715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1261.519840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1261.548231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1261.601139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1261.634159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1261.677721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1261.730161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1261.778362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1261.824973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1268.067693] LNet: 61608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.072924] LNetError: 61608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.082076] LNet: Removed LNI 192.168.204.22@tcp [ 1268.090917] LNet: Removed LNI 192.168.204.22@tcp1 [ 1268.447662] Key type .llcrypt unregistered [ 1268.451791] Key type ._llcrypt unregistered [ 1285.411951] Key type ._llcrypt registered [ 1285.417514] Key type .llcrypt registered [ 1286.672824] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 18:20:24 (1790979624) [ 1287.545397] Key type .llcrypt unregistered [ 1287.550785] Key type ._llcrypt unregistered [ 1294.666515] Key type ._llcrypt registered [ 1294.670096] Key type .llcrypt registered [ 1294.986681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1295.048301] alg: No test for adler32 (adler32-zlib) [ 1295.977843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1296.004255] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1303.904107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 1304.852497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e remote_dropped -r 1 [ 1304.913818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e remote_dropped -r 1 [ 1304.966876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1304.999526] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1305.291378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1305.334306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1305.405196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1305.445311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1305.479697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1305.515785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.557690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.602945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.639576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.664377] LNet: 62858:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.695497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.734550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.797264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.850148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1305.901547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.930932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1305.974144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1306.617811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e remote_error -r 1 [ 1306.657420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e remote_error -r 1 [ 1306.693881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1306.712810] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1306.907406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1306.935793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1306.959266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1307.007209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1307.052453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1307.099685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1307.140893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1307.178304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1307.216061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1307.233846] LNet: 62970:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1307.260724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1307.286594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1307.322583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1307.362858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1307.409770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1307.447317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1307.498860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1308.051483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e remote_timeout -r 1 [ 1308.100778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e remote_timeout -r 1 [ 1308.159253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1308.191578] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1308.451322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1308.477545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1308.505696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1308.546116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1308.576855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1308.606185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1308.639449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1308.681432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1308.734359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1308.754223] LNet: 63081:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1308.775180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1308.818233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1308.855061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1308.885629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1308.926677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1308.954628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1308.995798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1309.287754] LNet: 63122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.297748] LNetError: 63122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1309.319665] LNet: Removed LNI 192.168.204.22@tcp [ 1309.672968] Key type .llcrypt unregistered [ 1309.677468] Key type ._llcrypt unregistered [ 1324.856360] Key type ._llcrypt registered [ 1324.859235] Key type .llcrypt registered [ 1326.201877] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 18:21:03 (1790979663) [ 1327.159284] Key type .llcrypt unregistered [ 1327.161460] Key type ._llcrypt unregistered [ 1334.516465] Key type ._llcrypt registered [ 1334.519927] Key type .llcrypt registered [ 1334.906111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1334.943777] alg: No test for adler32 (adler32-zlib) [ 1335.854584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1335.875759] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1344.988665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 1354.329746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1354.397375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1354.425289] LNet: Added LNI 192.168.204.22@tcp1 [8/256/0/180] [ 1355.238976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 1355.819297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e remote_dropped -r 1 [ 1355.854509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp1 -m GET -e remote_dropped -r 1 [ 1355.887626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e remote_dropped -r 1 [ 1355.921529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -m GET -e remote_dropped -r 1 [ 1355.949717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp -m GET -e remote_dropped -r 1 [ 1355.982193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp1 -m GET -e remote_dropped -r 1 [ 1356.017833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -m GET -e remote_dropped -r 1 [ 1356.072119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -m GET -e remote_dropped -r 1 [ 1356.131391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1356.166371] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1356.170694] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.122@tcp1 [ 1356.510530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1356.549188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1356.583270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1356.622630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1356.682891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1356.744789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1356.806157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1356.865538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1356.949616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1356.984508] LNet: 64596:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1357.014287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1357.087248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1357.146988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1357.195269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1357.217224] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1357.232607] LNet: Skipped 3 previous similar messages [ 1357.256589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1357.305463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1357.343389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1357.381531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1357.823524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e remote_error -r 1 [ 1357.892588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp1 -m GET -e remote_error -r 1 [ 1357.955715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e remote_error -r 1 [ 1358.011903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -m GET -e remote_error -r 1 [ 1358.081535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp -m GET -e remote_error -r 1 [ 1358.149305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp1 -m GET -e remote_error -r 1 [ 1358.181886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -m GET -e remote_error -r 1 [ 1358.233670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -m GET -e remote_error -r 1 [ 1358.272173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1358.320317] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1358.569634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1358.623958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1358.651501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1358.680623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1358.717236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1358.754258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1358.782143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1358.822287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1358.884870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1358.915608] LNet: 64722:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1358.936126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1358.987594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1359.033349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1359.065689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1359.103620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1359.152940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1359.191890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1359.240421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1359.807708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e remote_timeout -r 1 [ 1359.873956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp1 -m GET -e remote_timeout -r 1 [ 1359.942348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e remote_timeout -r 1 [ 1360.020039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -m GET -e remote_timeout -r 1 [ 1360.073690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp -m GET -e remote_timeout -r 1 [ 1360.123214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp1 -m GET -e remote_timeout -r 1 [ 1360.176933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -m GET -e remote_timeout -r 1 [ 1360.244858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -m GET -e remote_timeout -r 1 [ 1360.309783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.122@tcp [ 1360.351052] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1360.584335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1360.613641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1360.638597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1360.661757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1360.691598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1360.717396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1360.743750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1360.765635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1360.788944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1360.804382] LNet: 64847:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1360.822285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1360.853959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1360.894706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1360.942334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1360.982321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1361.021119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1361.069418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1361.106661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1365.030742] LNet: 64925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.040842] LNetError: 64925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1365.051830] LNet: Removed LNI 192.168.204.22@tcp [ 1365.076862] LNet: Removed LNI 192.168.204.22@tcp1 [ 1365.438816] Key type .llcrypt unregistered [ 1365.441064] Key type ._llcrypt unregistered [ 1379.826103] Key type ._llcrypt registered [ 1379.827952] Key type .llcrypt registered [ 1381.060916] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 18:21:58 (1790979718) [ 1382.602444] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1384.172657] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 18:22:01 (1790979721) [ 1385.099307] Key type .llcrypt unregistered [ 1385.102247] Key type ._llcrypt unregistered [ 1391.544240] Key type ._llcrypt registered [ 1391.546874] Key type .llcrypt registered [ 1391.726087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1391.741831] alg: No test for adler32 (adler32-zlib) [ 1392.698954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1392.752068] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1400.857212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 1401.754448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e network_timeout -r 1 [ 1401.799310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e network_timeout -r 1 [ 1401.858179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 1401.881346] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1402.017065] LNet: 1 local NIs in recovery (showing 1): 192.168.204.22@tcp [ 1402.122157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1402.153417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1402.189077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1402.213903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1402.245212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1402.286490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1402.312685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1402.345968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1402.395664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1402.415392] LNet: 66469:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1402.444368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1402.472938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1402.505758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1402.553959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1402.589355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1402.630633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1402.901960] LNet: 66508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1402.907830] LNetError: 66508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1402.929607] LNet: Removed LNI 192.168.204.22@tcp [ 1403.281455] Key type .llcrypt unregistered [ 1403.284157] Key type ._llcrypt unregistered [ 1420.658424] Key type ._llcrypt registered [ 1420.661053] Key type .llcrypt registered [ 1420.966765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1420.991820] alg: No test for adler32 (adler32-zlib) [ 1421.967947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1421.996278] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1429.617898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 1439.510459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1439.565381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1439.597868] LNet: Added LNI 192.168.204.22@tcp1 [8/256/0/180] [ 1440.605290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 1441.133205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e network_timeout -r 1 [ 1441.186072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp1 -m GET -e network_timeout -r 1 [ 1441.259082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e network_timeout -r 1 [ 1441.305841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -m GET -e network_timeout -r 1 [ 1441.380877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp -m GET -e network_timeout -r 1 [ 1441.446646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp1 -m GET -e network_timeout -r 1 [ 1441.499884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -m GET -e network_timeout -r 1 [ 1441.543187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -m GET -e network_timeout -r 1 [ 1441.605348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 1441.632252] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1441.896373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1441.921551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1441.961945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1442.010236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1442.051461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1442.084784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1442.132627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1442.177850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1442.227877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1442.251102] LNet: 67395:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1442.265906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1442.298320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1442.322592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1442.354728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1442.378320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1442.405390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1446.853921] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1447.217040] LNet: 67470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.234800] LNetError: 67470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1447.240186] LNet: Removed LNI 192.168.204.22@tcp [ 1447.252978] LNet: Removed LNI 192.168.204.22@tcp1 [ 1447.595614] Key type .llcrypt unregistered [ 1447.599426] Key type ._llcrypt unregistered [ 1462.326414] Key type ._llcrypt registered [ 1462.328416] Key type .llcrypt registered [ 1463.488903] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 18:23:21 (1790979801) [ 1463.947983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1463.960412] alg: No test for adler32 (adler32-zlib) [ 1464.864518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.935259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1464.970195] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1465.670665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1465.711227] LNet: Added LNI 192.168.204.22@tcp1 [8/256/0/180] [ 1466.366631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 1466.421305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1466.470490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1466.511791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1466.542665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1466.591962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1466.643239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.22@tcp [ 1466.669333] LNetError: 68572:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.204.22@tcp to recovery [ 1466.975290] LNet: 1 local NIs in recovery (showing 1): 192.168.204.22@tcp [ 1466.975326] LNet: There was an unexpected network error while writing to 192.168.204.22: rc = -22 [ 1469.027587] LNet: There was an unexpected network error while writing to 192.168.204.22: rc = -22 [ 1470.184360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1473.120387] LNet: There was an unexpected network error while writing to 192.168.204.22: rc = -22 [ 1473.660642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1473.714324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1473.792275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1473.823215] LNet: 68679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1473.833773] LNetError: 68679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1473.845221] LNet: Removed LNI 192.168.204.22@tcp [ 1473.857606] LNet: Removed LNI 192.168.204.22@tcp1 [ 1473.898213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1473.955395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1473.985953] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1474.596823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1474.613670] LNet: Added LNI 192.168.204.22@tcp1 [8/256/0/180] [ 1475.584350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 1475.622522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1475.685530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1475.748372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1475.797857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1475.848552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1475.898587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1475.943364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.22@tcp [ 1475.977230] LNetError: 68952:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.204.22@tcp to recovery [ 1476.002592] LNet: There was an unexpected network error while writing to 192.168.204.22: rc = -22 [ 1478.366495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1482.143789] LNet: There was an unexpected network error while writing to 192.168.204.22: rc = -22 [ 1482.150552] LNet: Skipped 1 previous similar message [ 1487.435586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1487.475747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1487.519670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1493.913112] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 18:23:51 (1790979831) [ 1494.194844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1494.228651] LNet: 69669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.235556] LNetError: 69669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.241173] LNetError: 69669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1494.246771] LNet: Removed LNI 192.168.204.22@tcp [ 1494.263758] LNet: Removed LNI 192.168.204.22@tcp1 [ 1494.299657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.373337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1494.406255] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1495.139808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1495.157813] LNet: Added LNI 192.168.204.22@tcp1 [8/256/0/180] [ 1496.145076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 1496.266374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1496.301964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1496.340397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1496.383671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1496.433979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1496.513198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.22@tcp --health 0 [ 1496.678609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1497.439407] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.22@tcp [ 1497.444800] LNet: There was an unexpected network error while writing to 192.168.204.22: rc = -22 [ 1497.464271] LNet: Skipped 1 previous similar message [ 1512.098601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1512.296384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1512.347091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1512.407202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.22@tcp --health 500 [ 1512.491726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1519.182589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1519.247592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1519.281662] LNet: 70161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.290383] LNetError: 70161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1519.297663] LNetError: 70161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1519.307634] LNet: Removed LNI 192.168.204.22@tcp [ 1519.321900] LNet: Removed LNI 192.168.204.22@tcp1 [ 1519.406315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.479695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1519.508147] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1520.222307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1520.241785] LNet: Added LNI 192.168.204.22@tcp1 [8/256/0/180] [ 1520.951240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 1520.996722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1521.066750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1521.107183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1521.164311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1521.203155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1521.246500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1521.297175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.22@tcp --health 0 [ 1522.527846] LNet: There was an unexpected network error while writing to 192.168.204.22: rc = -22 [ 1522.533912] LNet: Skipped 3 previous similar messages [ 1523.672141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1533.855023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1533.892817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1533.929227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1540.848131] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 18:24:38 (1790979878) [ 1541.478522] LNet: 71221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.486831] LNetError: 71221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1541.492956] LNetError: 71221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1541.499504] LNet: Removed LNI 192.168.204.22@tcp [ 1541.519828] LNet: Removed LNI 192.168.204.22@tcp1 [ 1542.232733] Key type .llcrypt unregistered [ 1542.234208] Key type ._llcrypt unregistered [ 1548.430634] Key type ._llcrypt registered [ 1548.432704] Key type .llcrypt registered [ 1548.773677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1548.801900] alg: No test for adler32 (adler32-zlib) [ 1549.740896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1549.764735] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1558.301874] Lustre: DEBUG MARKER: Initial discovery [ 1558.401838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.122@tcp [ 1560.699404] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1561.057259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1561.129770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.122@tcp [ 1561.179803] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1561.211137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1561.254045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.122@tcp -r 1 -m GET -l 3 [ 1561.296692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.122@tcp -r 1 -m GET -e local_error [ 1561.375290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.122@tcp -r 1 -m PUT -l 6 [ 1562.080257] LNet: 1 local NIs in recovery (showing 1): 192.168.204.22@tcp [ 1562.795078] Lustre: DEBUG MARKER: Force oleg422-client.virtnet to discover 192.168.204.122@tcp (in background) [ 1562.853219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.122@tcp [ 1564.346940] Lustre: DEBUG MARKER: Force oleg422-server to discover 192.168.204.22@tcp [ 1565.920355] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1567.239756] Lustre: DEBUG MARKER: Wait for 71974 [ 1568.990543] Lustre: DEBUG MARKER: Finished wait on 71974 [ 1569.081380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1569.118537] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1569.138834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1569.481938] LNet: 72432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.486828] LNetError: 72432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.504472] LNet: Removed LNI 192.168.204.22@tcp [ 1569.994872] Key type .llcrypt unregistered [ 1570.000404] Key type ._llcrypt unregistered [ 1588.534409] Key type ._llcrypt registered [ 1588.537324] Key type .llcrypt registered [ 1589.845824] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 18:25:27 (1790979927) [ 1590.625352] Key type .llcrypt unregistered [ 1590.627772] Key type ._llcrypt unregistered [ 1596.665346] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1596.917088] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1596.921986] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1596.928289] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1599.445721] Key type ._llcrypt registered [ 1599.450888] Key type .llcrypt registered [ 1599.643730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1599.656488] alg: No test for adler32 (adler32-zlib) [ 1600.537716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.623644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1600.659702] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1601.307303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1601.322324] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1608.207152] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 18:25:45 (1790979945) [ 1608.360227] LNet: 74503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.372644] LNetError: 74503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1608.380047] LNet: Removed LNI 10.1.2.3@tcp [ 1608.391145] LNet: Removed LNI 192.168.204.22@tcp [ 1609.030113] Key type .llcrypt unregistered [ 1609.039509] Key type ._llcrypt unregistered [ 1615.709298] Key type ._llcrypt registered [ 1615.711518] Key type .llcrypt registered [ 1615.899361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1615.916544] alg: No test for adler32 (adler32-zlib) [ 1616.845708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.898408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1616.919593] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1617.594980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1617.611928] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1618.623614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.22@tcp [ 1619.679520] LNetError: Unexpected error -22 connecting to 192.168.204.22@tcp at host 192.168.204.22:988 [ 1624.341980] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 18:26:02 (1790979962) [ 1624.578809] LNet: 75639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.597803] LNetError: 75639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1625.631159] LNet: Removed LNI 10.1.2.3@tcp [ 1625.636328] LNetError: 75639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1625.653981] LNet: Removed LNI 192.168.204.22@tcp [ 1626.345831] Key type .llcrypt unregistered [ 1626.347991] Key type ._llcrypt unregistered [ 1632.287270] Key type ._llcrypt registered [ 1632.289713] Key type .llcrypt registered [ 1632.527963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1632.542664] alg: No test for adler32 (adler32-zlib) [ 1633.453316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.537344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1633.576303] LNet: Added LNI 192.168.204.22@tcp1 [8/256/0/180] [ 1634.278649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1634.309475] LNet: Added LNI 192.168.204.22@tcp2 [8/256/0/180] [ 1635.206438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.22@tcp1 --nid 192.168.204.22@tcp2 [ 1639.538258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.22@tcp1 192.168.204.22@tcp2 [ 1639.563409] LNetError: 76473:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.22@tcp1 to 192.168.204.22@tcp2 - Route table may be misconfigured [ 1639.586989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.22@tcp2 192.168.204.22@tcp1 [ 1645.454398] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 18:26:23 (1790979983) [ 1645.548106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.584081] LNet: 77040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.600576] LNetError: 77040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1646.623863] LNet: Removed LNI 192.168.204.22@tcp1 [ 1646.625991] LNetError: 77040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1646.653364] LNet: Removed LNI 192.168.204.22@tcp2 [ 1646.707798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.782860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1646.826746] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1647.469832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1647.503312] LNet: Added LNI 192.168.204.22@tcp1 [8/256/0/180] [ 1648.446801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 1648.476911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.22@tcp 192.168.204.22@tcp [ 1648.512522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.22@tcp1 192.168.204.22@tcp1 [ 1648.549341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.22@tcp -d 192.168.204.22@tcp -e network_timeout [ 1648.574303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -e network_timeout [ 1648.604559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -e network_timeout [ 1648.630869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -e network_timeout [ 1648.657907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 1648.673880] LNet: There was an unexpected network error while writing to 192.168.204.22: rc = -22 [ 1648.688863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1648.770945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1648.802276] LNet: 1 local NIs in recovery (showing 1): 192.168.204.22@tcp [ 1648.854260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1654.791492] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 18:26:32 (1790979992) [ 1655.099435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.124291] LNet: 77908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.145136] LNetError: 77908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.153508] LNet: Removed LNI 192.168.204.22@tcp [ 1656.166355] LNet: Removed LNI 192.168.204.22@tcp1 [ 1656.201494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.328866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1656.559983] LNet: 77945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1656.997883] Key type .llcrypt unregistered [ 1657.002372] Key type ._llcrypt unregistered [ 1668.257418] Key type ._llcrypt registered [ 1668.259796] Key type .llcrypt registered [ 1669.536299] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 18:26:47 (1790980007) [ 1669.824139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1669.837936] alg: No test for adler32 (adler32-zlib) [ 1670.827356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.509969] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1674.269595] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 18:26:51 (1790980011) [ 1674.504958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1674.547554] LNet: 79040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.596811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1674.708810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1674.755241] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1675.460913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1675.477781] LNet: Added LNI 192.168.204.22@tcp1 [8/256/0/180] [ 1676.660196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 1676.701861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp1 [ 1676.745331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp1 [ 1683.409614] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 18:27:00 (1790980020) [ 1685.571360] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg422-server [ 1687.290339] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 18:27:04 (1790980024) [ 1689.473797] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg422-server [ 1690.984674] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 18:27:08 (1790980028) [ 1693.386955] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg422-server [ 1695.706565] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 18:27:12 (1790980032) [ 1697.803662] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg422-server [ 1699.509962] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 18:27:17 (1790980037) [ 1702.150949] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg422-server [ 1704.130799] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 18:27:21 (1790980041) [ 1705.983803] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg422-server [ 1707.930451] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 18:27:25 (1790980045) [ 1710.210435] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg422-server [ 1712.095677] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 18:27:29 (1790980049) [ 1714.156738] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg422-server [ 1716.079252] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 18:27:33 (1790980053) [ 1718.017353] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg422-server [ 1719.985157] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 18:27:37 (1790980057) [ 1720.123484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.173164] LNet: 82915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.186990] LNetError: 82915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.197496] LNet: Removed LNI 192.168.204.22@tcp [ 1721.256742] LNet: Removed LNI 192.168.204.22@tcp1 [ 1721.305913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.367554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1721.397977] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1722.213747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1722.326357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 1722.520812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.553394] LNet: 83068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.560614] LNetError: 83068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.574833] LNetError: 83068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1722.599598] LNet: Removed LNI 192.168.204.22@tcp [ 1722.662948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.777131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1722.850379] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1723.781491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 1724.045300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.065819] LNetError: 83213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1725.097894] LNet: Removed LNI 192.168.204.22@tcp [ 1725.136062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.274926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.368997] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1726.863678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1726.998235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 1727.368739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.424280] LNet: 83361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.426975] LNet: 83361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1727.448366] LNetError: 83361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1728.490532] LNet: Removed LNI 192.168.204.22@tcp [ 1728.559868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.659888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1728.729140] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1730.245942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 1730.556484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.624231] LNet: Removed LNI 192.168.204.22@tcp [ 1731.670655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.750792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1731.783331] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1732.946311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1733.073339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 1733.444467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.492740] LNet: 83654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.502303] LNet: 83654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1733.518226] LNetError: 83654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.530065] LNetError: 83654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1733.559092] LNet: Removed LNI 192.168.204.22@tcp [ 1733.609687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.738524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1733.788484] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1734.531182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 1734.872901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.984973] LNet: Removed LNI 192.168.204.22@tcp [ 1736.037160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.123459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1736.170828] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1737.428631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1737.542225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 1737.875464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.985091] LNet: Removed LNI 192.168.204.22@tcp [ 1739.006038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.095676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1739.131535] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1740.152258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 1741.744138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.782740] LNet: 84101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.795536] LNet: 84101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1741.804259] LNetError: 84101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.811312] LNetError: 84101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1741.830747] LNet: Removed LNI 192.168.204.22@tcp [ 1741.889430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.955976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1742.000592] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1743.111513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1743.248352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 1744.706661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.769098] LNet: Removed LNI 192.168.204.22@tcp [ 1745.797863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.900343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1745.939043] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1746.755950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 1747.121785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.197947] LNet: Removed LNI 192.168.204.22@tcp [ 1748.222538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.306930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1748.369343] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1749.285878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1749.421581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 1749.705703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1750.767581] LNet: Removed LNI 192.168.204.22@tcp [ 1750.845821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.903959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1750.929359] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1751.913784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 1753.484514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1754.534776] LNet: Removed LNI 192.168.204.22@tcp [ 1754.579789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.668089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1754.717230] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1755.714660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1755.788643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.844562] LNet: Removed LNI 192.168.204.22@tcp [ 1755.903340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.999668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1756.033151] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1757.185800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1757.320948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.376946] LNet: Removed LNI 192.168.204.22@tcp [ 1757.416608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.495747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1757.595537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.657717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.721733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1757.843267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.858635] LNet: 85002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.864826] LNet: 85002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 1757.881911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.963531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1758.076289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.156133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.220610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1758.243127] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1759.736833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.777038] LNetError: 85146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.783253] LNetError: 85146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1759.809426] LNet: Removed LNI 192.168.204.22@tcp [ 1759.846125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.020415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1769.730549] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 18:28:27 (1790980107) [ 1769.968371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1770.023469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.126669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1770.181862] LNet: Added LNI 192.168.204.22@tcp231 [8/256/0/180] [ 1771.916666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.971374] LNet: Removed LNI 192.168.204.22@tcp231 [ 1772.011382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.151855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1772.230854] LNet: Added LNI 192.168.204.22@tcp231 [8/256/0/180] [ 1772.243170] LNet: 85875:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1773.123808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1773.164629] LNet: Removed LNI 192.168.204.22@tcp231 [ 1773.184151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1773.234528] LNet: Added LNI 192.168.204.22@tcp231 [8/256/0/180] [ 1774.750575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.795684] LNet: Removed LNI 192.168.204.22@tcp231 [ 1774.847490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.956926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1775.007385] LNet: Added LNI 192.168.204.22@tcp231 [8/256/0/0] [ 1775.021399] LNet: 86028:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1775.681312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1775.731789] LNet: Removed LNI 192.168.204.22@tcp231 [ 1775.757158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1775.790399] LNet: Added LNI 192.168.204.22@tcp231 [8/256/0/0] [ 1777.505971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1777.551694] LNet: Removed LNI 192.168.204.22@tcp231 [ 1777.582572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.693733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1777.718778] LNet: Added LNI 192.168.204.22@tcp231 [8/256/0/60] [ 1777.722841] LNet: 86180:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1777.971761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1778.000661] LNet: Removed LNI 192.168.204.22@tcp231 [ 1778.025675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1778.047220] LNet: Added LNI 192.168.204.22@tcp231 [8/256/0/60] [ 1779.302560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.362110] LNet: Removed LNI 192.168.204.22@tcp231 [ 1779.384897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.496358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1779.527182] LNet: Added LNI 192.168.204.22@tcp231 [8/256/0/180] [ 1779.965229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1779.996854] LNet: Removed LNI 192.168.204.22@tcp231 [ 1780.024209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1780.055699] LNet: Added LNI 192.168.204.22@tcp231 [8/256/0/180] [ 1787.925340] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 18:28:45 (1790980125) [ 1788.057329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.122112] LNet: Removed LNI 192.168.204.22@tcp231 [ 1788.164516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1788.248987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1788.288848] LNet: Added LNI 192.168.204.22@tcp232 [8/256/0/180] [ 1789.117888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1789.176527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1789.228082] LNet: Removed LNI 192.168.204.22@tcp232 [ 1789.255415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1789.302225] LNet: Added LNI 192.168.204.22@tcp232 [8/256/0/180] [ 1790.060644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1790.112112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1790.142288] LNet: Removed LNI 192.168.204.22@tcp232 [ 1790.182367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1790.223347] LNet: Added LNI 192.168.204.22@tcp232 [8/256/0/180] [ 1791.038763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1791.071891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1791.096578] LNet: Removed LNI 192.168.204.22@tcp232 [ 1791.108565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1791.137199] LNet: Added LNI 192.168.204.22@tcp232 [8/256/0/180] [ 1791.845376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1791.897941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1791.931680] LNet: Removed LNI 192.168.204.22@tcp232 [ 1791.947884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1798.642510] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 18:28:56 (1790980136) [ 1799.025743] LNet: 88170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.030432] LNet: 88170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1799.679776] Key type .llcrypt unregistered [ 1799.688530] Key type ._llcrypt unregistered [ 1806.234630] Key type ._llcrypt registered [ 1806.238651] Key type .llcrypt registered [ 1806.495297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1806.512919] alg: No test for adler32 (adler32-zlib) [ 1807.617901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1807.682914] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1815.241484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 1824.193765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1824.235475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1824.265377] LNet: Added LNI 192.168.204.22@tcp1 [8/256/0/180] [ 1825.323614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 1826.182975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_interrupt -r 2 [ 1826.244708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp1 -m GET -e local_interrupt -r 2 [ 1826.294192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_interrupt -r 2 [ 1826.344233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -m GET -e local_interrupt -r 2 [ 1826.402439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp -m GET -e local_interrupt -r 2 [ 1826.458399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp1 -m GET -e local_interrupt -r 2 [ 1826.512249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -m GET -e local_interrupt -r 2 [ 1826.560543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -m GET -e local_interrupt -r 2 [ 1826.595632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.22@tcp 192.168.204.122@tcp [ 1826.633964] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1826.645857] LNet: 1 local NIs in recovery (showing 1): 192.168.204.22@tcp [ 1826.866986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1826.905310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1826.946852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1827.012110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1827.061437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1827.116882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1827.167266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1827.203866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1827.257551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1827.278178] LNet: 88838:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1827.323539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1827.361991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1827.403440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1827.436216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1827.486222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1827.540506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1828.010235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_dropped -r 2 [ 1828.039558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp1 -m GET -e local_dropped -r 2 [ 1828.067599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_dropped -r 2 [ 1828.110364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -m GET -e local_dropped -r 2 [ 1828.141383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp -m GET -e local_dropped -r 2 [ 1828.177677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp1 -m GET -e local_dropped -r 2 [ 1828.229975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -m GET -e local_dropped -r 2 [ 1828.287964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -m GET -e local_dropped -r 2 [ 1828.339901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.22@tcp 192.168.204.122@tcp [ 1828.387803] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1828.636324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1828.664258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1828.704423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1828.745497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1828.794683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1828.851387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1828.912646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1828.949737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1828.977863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1828.993704] LNet: 88961:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1829.007530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1829.037964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1829.080167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1829.126310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1829.166269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1829.216834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1829.675284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_aborted -r 2 [ 1829.728206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp1 -m GET -e local_aborted -r 2 [ 1829.764808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_aborted -r 2 [ 1829.804398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -m GET -e local_aborted -r 2 [ 1829.841950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp -m GET -e local_aborted -r 2 [ 1829.881722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp1 -m GET -e local_aborted -r 2 [ 1829.956540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -m GET -e local_aborted -r 2 [ 1830.011608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -m GET -e local_aborted -r 2 [ 1830.067226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.22@tcp 192.168.204.122@tcp [ 1830.101956] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1830.363332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1830.384406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1830.412994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1830.437776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1830.465739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1830.503939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1830.532338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1830.567985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1830.605209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1830.623480] LNet: 89083:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1830.641965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1830.673510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1830.714049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1830.758066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1830.802235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1830.842755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1831.282983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_no_route -r 2 [ 1831.313948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp1 -m GET -e local_no_route -r 2 [ 1831.336766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_no_route -r 2 [ 1831.367811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -m GET -e local_no_route -r 2 [ 1831.401966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp -m GET -e local_no_route -r 2 [ 1831.426269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp1 -m GET -e local_no_route -r 2 [ 1831.458905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -m GET -e local_no_route -r 2 [ 1831.494422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -m GET -e local_no_route -r 2 [ 1831.520217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.22@tcp 192.168.204.122@tcp [ 1831.721516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1831.745604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1831.777041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1831.804117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1831.829232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1831.868218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1831.916042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1831.966052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1832.027494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1832.074691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1832.124471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1832.151770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1832.194348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1832.220353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1832.261167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1832.791293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e local_timeout -r 2 [ 1832.825134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp1 -m GET -e local_timeout -r 2 [ 1832.863817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e local_timeout -r 2 [ 1832.895689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -m GET -e local_timeout -r 2 [ 1832.927880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp -m GET -e local_timeout -r 2 [ 1832.961968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp1 -m GET -e local_timeout -r 2 [ 1832.998657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -m GET -e local_timeout -r 2 [ 1833.037755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -m GET -e local_timeout -r 2 [ 1833.106700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.22@tcp 192.168.204.122@tcp [ 1833.141558] LNet: There was an unexpected network error while writing to 192.168.204.122: rc = -22 [ 1833.147916] LNet: Skipped 1 previous similar message [ 1833.407823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1833.433511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1833.463691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1833.495828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1833.520808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1833.551775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1833.590605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1833.624730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1833.670045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1833.693887] LNet: 89328:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1833.705157] LNet: 89328:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1833.745459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1833.792259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1833.835156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1833.872110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1833.915341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1833.974820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1834.503564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp -m GET -e remote_dropped -r 2 [ 1834.556099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.122@tcp1 -m GET -e remote_dropped -r 2 [ 1834.598781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp -m GET -e remote_dropped -r 2 [ 1834.644438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp -d 192.168.204.22@tcp1 -m GET -e remote_dropped -r 2 [ 1834.700056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp -m GET -e remote_dropped -r 2 [ 1834.737120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.22@tcp1 -d 192.168.204.122@tcp1 -m GET -e remote_dropped -r 2 [ 1834.769848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp -m GET -e remote_dropped -r 2 [ 1834.847360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.22@tcp1 -d 192.168.204.22@tcp1 -m GET -e remote_dropped -r 2 [ 1834.907182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.22@tcp 192.168.204.122@tcp [ 1834.946382] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.122@tcp [ 1835.147553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1835.182029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1835.222196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1835.278720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1835.330681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1835.384364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1835.459669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1835.500850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1835.533413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1835.568911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1835.599816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1835.624288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1835.658035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1835.692158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1835.727095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1840.540926] LNet: 89524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.557282] LNetError: 89524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.567304] LNet: Removed LNI 192.168.204.22@tcp [ 1840.591487] LNet: Removed LNI 192.168.204.22@tcp1 [ 1841.065751] Key type .llcrypt unregistered [ 1841.069648] Key type ._llcrypt unregistered [ 1857.548375] Key type ._llcrypt registered [ 1857.551402] Key type .llcrypt registered [ 1858.680735] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 18:29:56 (1790980196) [ 1859.200287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1859.211460] alg: No test for adler32 (adler32-zlib) [ 1860.149715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1860.209686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1860.231952] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1860.873776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1860.894573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.22@tcp [ 1860.914945] LNetError: 90503:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.204.22@tcp to recovery [ 1861.011385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.22@tcp [ 1861.142484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1866.945967] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 18:30:04 (1790980204) [ 1867.120300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1867.154388] LNet: 91089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1867.174123] LNetError: 91089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1867.213359] LNet: Removed LNI 192.168.204.22@tcp [ 1867.244061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1867.324196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1867.422924] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1868.109379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1868.127687] LNet: Added LNI 192.168.204.22@tcp1 [8/256/0/180] [ 1868.982441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.22@tcp [ 1869.007423] LNetError: 91358:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.204.22@tcp to recovery [ 1869.343480] LNet: 1 local NIs in recovery (showing 1): 192.168.204.22@tcp [ 1869.347100] LNet: There was an unexpected network error while writing to 192.168.204.22: rc = -22 [ 1870.296240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.22@tcp [ 1878.610431] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 18:30:15 (1790980215) [ 1880.773284] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg422-server [ 1883.061821] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 18:30:20 (1790980220) [ 1885.229498] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg422-server [ 1887.984841] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 18:30:24 (1790980224) [ 1889.572123] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg422-server [ 1892.071922] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 18:30:29 (1790980229) [ 1892.381306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1892.452219] LNet: 92981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1892.466755] LNetError: 92981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1892.488291] LNet: Removed LNI 192.168.204.22@tcp [ 1892.503866] LNet: Removed LNI 192.168.204.22@tcp1 [ 1892.552206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1892.669669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1892.702897] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1893.632831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1893.695879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1893.744948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1893.856379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1893.897173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1893.981834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1901.287077] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 18:30:38 (1790980238) [ 1901.535305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1901.576703] LNet: 93713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.581854] LNetError: 93713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.593629] LNetError: 93713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1901.612041] LNet: Removed LNI 192.168.204.22@tcp [ 1901.649627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.735827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1901.756995] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1909.703312] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 18:30:46 (1790980246) [ 1909.834628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.867230] LNet: 94410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.876719] LNetError: 94410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1909.914205] LNet: Removed LNI 192.168.204.22@tcp [ 1909.947735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1910.075926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1910.128688] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1918.649522] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 18:30:55 (1790980255) [ 1920.038403] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1921.934816] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 18:30:59 (1790980259) [ 1922.566438] LNet: 95451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.589584] LNetError: 95451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.606132] LNet: Removed LNI 192.168.204.22@tcp [ 1923.306243] Key type .llcrypt unregistered [ 1923.309976] Key type ._llcrypt unregistered [ 1930.041099] Key type ._llcrypt registered [ 1930.047183] Key type .llcrypt registered [ 1930.300413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1930.335852] alg: No test for adler32 (adler32-zlib) [ 1931.236326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1931.267298] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1938.917425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 1944.834503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.122@tcp [ 1945.266685] LNet: 95867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.273720] LNetError: 95867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.303778] LNet: Removed LNI 192.168.204.22@tcp [ 1945.799889] Key type .llcrypt unregistered [ 1945.802636] Key type ._llcrypt unregistered [ 1958.612322] Key type ._llcrypt registered [ 1958.616506] Key type .llcrypt registered [ 1960.396280] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1962.098394] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1964.307703] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 18:31:41 (1790980301) [ 1965.905940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1967.752784] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 18:31:45 (1790980305) [ 1969.711278] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg422-server [ 1971.650461] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 18:31:49 (1790980309) [ 1973.578312] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg422-server [ 1975.190175] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 18:31:52 (1790980312) [ 1976.970758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1977.076997] alg: No test for adler32 (adler32-zlib) [ 1978.306650] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 1978.556156] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 1980.255320] Key type lgssc registered [ 1981.973524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2003.216983] Key type lgssc unregistered [ 2003.451156] LNet: 99049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.467943] LNetError: 99049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2003.487351] LNet: Removed LNI 192.168.204.22@tcp [ 2004.138911] Key type .llcrypt unregistered [ 2004.140840] Key type ._llcrypt unregistered [ 2021.057346] Key type ._llcrypt registered [ 2021.060614] Key type .llcrypt registered [ 2022.484162] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 18:32:39 (1790980359) [ 2023.021853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2023.036957] alg: No test for adler32 (adler32-zlib) [ 2023.986778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2024.058755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2024.139361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2024.215390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2024.281181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2031.893800] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 18:32:49 (1790980369) [ 2032.093053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2032.120265] LNet: 100471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.152159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.234915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2032.271831] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2033.269700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2033.288583] LNet: 100616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.294782] LNetError: 100616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2033.313529] LNet: Removed LNI 192.168.204.22@tcp [ 2033.341117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.387389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2033.406474] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2034.010973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2034.051917] LNetError: 100644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2034.072838] LNet: Removed LNI 192.168.204.22@tcp [ 2034.087955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2034.141348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2034.168345] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2041.394161] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 18:32:58 (1790980378) [ 2041.506033] LNet: 101226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.512313] LNet: 101226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2041.519534] LNetError: 101226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.545585] LNet: Removed LNI 192.168.204.22@tcp [ 2042.180911] Key type .llcrypt unregistered [ 2042.183312] Key type ._llcrypt unregistered [ 2050.149797] Key type ._llcrypt registered [ 2050.152104] Key type .llcrypt registered [ 2050.440672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2050.458884] alg: No test for adler32 (adler32-zlib) [ 2051.436877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2051.546564] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2051.552992] LNet: *** cfs_fail_loc=e003, val=100*** [ 2052.414792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2052.447537] LNet: *** cfs_fail_loc=e003, val=100*** [ 2052.449683] LNet: Skipped 2 previous similar messages [ 2052.468823] Ping reply received with mapping: [ 2052.471038] found metadata reply NI[192.168.204.22@tcp] [ 2052.474948] 16 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2052.481122] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2052.487946] 5a 5a 5a 5a 5a 5a 5a 5a [ 2052.645624] LNet: 101631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.653307] LNetError: 101631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2053.675789] LNet: Removed LNI 192.168.204.22@tcp [ 2054.650982] Key type .llcrypt unregistered [ 2054.657445] Key type ._llcrypt unregistered [ 2067.484973] Key type ._llcrypt registered [ 2067.488737] Key type .llcrypt registered [ 2068.845266] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 18:33:26 (1790980406) [ 2069.437366] Key type .llcrypt unregistered [ 2069.441616] Key type ._llcrypt unregistered [ 2076.420279] Key type ._llcrypt registered [ 2076.422766] Key type .llcrypt registered [ 2076.747682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2076.778113] alg: No test for adler32 (adler32-zlib) [ 2077.982203] LNetError: 102671:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2078.468367] Key type .llcrypt unregistered [ 2078.471828] Key type ._llcrypt unregistered [ 2083.899604] Key type ._llcrypt registered [ 2083.909189] Key type .llcrypt registered [ 2085.501360] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 18:33:42 (1790980422) [ 2086.029341] Key type .llcrypt unregistered [ 2086.031322] Key type ._llcrypt unregistered [ 2093.093395] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2093.107726] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2095.718675] Key type ._llcrypt registered [ 2095.722607] Key type .llcrypt registered [ 2095.976738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2096.004259] alg: No test for adler32 (adler32-zlib) [ 2096.940319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.000157] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2106.764344] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 18:34:04 (1790980444) [ 2106.951975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2106.997297] LNet: 104454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2107.008755] LNetError: 104454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2107.035707] LNet: Removed LNI 10.1.2.3@tcp [ 2107.076783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.104868] Key type .llcrypt unregistered [ 2108.110501] Key type ._llcrypt unregistered [ 2115.368388] Key type ._llcrypt registered [ 2115.371671] Key type .llcrypt registered [ 2115.531836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2115.545559] alg: No test for adler32 (adler32-zlib) [ 2116.627147] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2116.785934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2117.110337] LNet: 104746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.119345] LNetError: 104746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2117.138249] LNet: Removed LNI 192.168.204.22@tcp [ 2118.242717] Key type .llcrypt unregistered [ 2118.248113] Key type ._llcrypt unregistered [ 2131.515326] Key type ._llcrypt registered [ 2131.517553] Key type .llcrypt registered [ 2132.865569] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 18:34:30 (1790980470) [ 2134.098614] Key type .llcrypt unregistered [ 2134.103874] Key type ._llcrypt unregistered [ 2142.336493] Key type ._llcrypt registered [ 2142.342448] Key type .llcrypt registered [ 2142.610790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2142.630374] alg: No test for adler32 (adler32-zlib) [ 2151.782593] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 18:34:48 (1790980488) [ 2151.905746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2151.964343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2152.085837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2152.113989] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2153.045162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2153.108963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2153.151640] LNet: Removed LNI 192.168.204.22@tcp [ 2153.410572] LNet: 106491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2153.828675] Key type .llcrypt unregistered [ 2153.834371] Key type ._llcrypt unregistered [ 2166.977349] Key type ._llcrypt registered [ 2166.980875] Key type .llcrypt registered [ 2168.683212] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 18:35:05 (1790980505) [ 2169.179615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2169.198964] alg: No test for adler32 (adler32-zlib) [ 2170.161443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.302765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2170.334680] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2171.196636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2171.258853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.22@tcp [ 2171.281228] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2171.285767] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2171.288521] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2171, 0, 0 [ 2171.292404] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2171.296131] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2171.302943] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2171.306704] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2171, 0, 0 [ 2171.313954] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2171.317846] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2171.322629] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2171.328887] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2171, 0, 2221 [ 2171.337030] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2171.341328] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2171.344868] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2171.347610] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2171, 0, 2221 [ 2171.354298] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2178.625646] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 18:35:15 (1790980515) [ 2179.246156] LNet: 108030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2179.253163] LNetError: 108030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2179.270452] LNet: Removed LNI 192.168.204.22@tcp [ 2180.031813] Key type .llcrypt unregistered [ 2180.039313] Key type ._llcrypt unregistered [ 2186.511422] Key type ._llcrypt registered [ 2186.513731] Key type .llcrypt registered [ 2186.666046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2186.674844] alg: No test for adler32 (adler32-zlib) [ 2187.558379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2187.594201] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2196.005966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 2206.618453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2206.698108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2206.724181] LNet: Added LNI 192.168.204.22@tcp1 [8/256/0/180] [ 2207.773748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 2208.032765] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2208.319658] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2208.328409] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2208.335279] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2209.696324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2209.732459] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2227.695758] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 18:36:04 (1790980564) [ 2227.838073] LNet: 109529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2227.847779] LNetError: 109529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2228.896133] LNet: Removed LNI 192.168.204.22@tcp [ 2228.907653] LNetError: 109529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2228.923478] LNet: Removed LNI 192.168.204.22@tcp1 [ 2228.933649] LNet: Removed LNI 10.1.2.3@tcp99 [ 2229.846736] Key type .llcrypt unregistered [ 2229.852747] Key type ._llcrypt unregistered [ 2236.983493] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2236.986061] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2239.636631] Key type ._llcrypt registered [ 2239.639884] Key type .llcrypt registered [ 2239.943408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2239.959959] alg: No test for adler32 (adler32-zlib) [ 2240.971725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.044964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2241.098190] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2242.044848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2242.070354] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2245.213088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2245.326247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2245.435746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2245.503269] LNetError: 110281:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2245.583653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2245.660699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2246.444237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2246.582334] LNet: 110301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2246.593894] LNetError: 110301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2246.603770] LNet: Removed LNI 10.1.2.3@tcp [ 2247.653790] LNet: Removed LNI 192.168.204.22@tcp [ 2248.404402] Key type .llcrypt unregistered [ 2248.411302] Key type ._llcrypt unregistered [ 2264.961831] Key type ._llcrypt registered [ 2264.970117] Key type .llcrypt registered [ 2267.101916] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 18:36:43 (1790980603) [ 2267.846614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2267.879627] alg: No test for adler32 (adler32-zlib) [ 2268.900753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2269.028584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2269.067747] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2270.126742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2270.218982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg422-client@tcp [ 2278.161478] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 18:36:55 (1790980615) [ 2278.235537] LNet: 111825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2278.239804] LNetError: 111825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2278.268487] LNet: Removed LNI 192.168.204.22@tcp [ 2279.089554] Key type .llcrypt unregistered [ 2279.093678] Key type ._llcrypt unregistered [ 2285.273639] Key type ._llcrypt registered [ 2285.284694] Key type .llcrypt registered [ 2285.551811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2285.575053] alg: No test for adler32 (adler32-zlib) [ 2286.543959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2286.601354] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2286.664243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2292.934371] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 18:37:10 (1790980630) [ 2294.431310] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2295.948667] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 18:37:13 (1790980633) [ 2297.513733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2299.368195] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 18:37:16 (1790980636) [ 2299.533234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2299.565104] LNet: 113254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2299.581054] LNetError: 113254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2299.615076] LNet: Removed LNI 192.168.204.22@tcp [ 2299.649666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.759257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2299.788314] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2300.575518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2300.659522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2308.232515] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 18:37:25 (1790980645) [ 2308.342507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2308.387343] LNet: 113957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2308.401690] LNetError: 113957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2308.430829] LNet: Removed LNI 192.168.204.22@tcp [ 2308.501245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2308.555322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2308.586840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2309.436312] Key type .llcrypt unregistered [ 2309.438104] Key type ._llcrypt unregistered [ 2322.451189] Key type ._llcrypt registered [ 2322.454680] Key type .llcrypt registered [ 2324.143877] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 18:37:41 (1790980661) [ 2324.700149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2324.720977] alg: No test for adler32 (adler32-zlib) [ 2325.604182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2325.645342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2325.665427] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2326.669800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2326.751912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2334.559529] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 18:37:51 (1790980671) [ 2334.890818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2334.926357] LNet: 115519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2334.937565] LNetError: 115519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2335.978555] LNet: Removed LNI 192.168.204.22@tcp [ 2336.073936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2336.204316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2336.271477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2343.688951] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 18:38:01 (1790980681) [ 2344.031293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2344.077388] LNet: 116100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.099578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2344.198808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2344.809995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2344.859365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2345.011285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2345.600345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2345.643830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2345.728040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2345.756799] LNet: 116133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2345.792524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2345.874701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2346.390556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2346.442098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2346.517392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2346.999916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2347.052341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2355.549962] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 18:38:12 (1790980692) [ 2355.920751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2355.969228] LNet: 116734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.993972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2356.088367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2356.131652] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2356.942628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2356.962119] LNet: Added LNI 192.168.204.22@tcp2 [8/256/0/180] [ 2357.915963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2357.965934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2358.244665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.276220] LNetError: 117013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2358.281610] LNet: Removed LNI 192.168.204.22@tcp [ 2358.295569] LNet: Removed LNI 192.168.204.22@tcp2 [ 2358.318819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.366587] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2358.373824] LNet: Added LNI 192.168.204.22@tcp2 [8/256/0/180] [ 2358.382545] LNet: 117019:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2359.151765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2359.382741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2359.405227] LNetError: 117056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2359.418043] LNetError: 117056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2360.479182] LNet: Removed LNI 192.168.204.22@tcp [ 2360.499434] LNet: Removed LNI 192.168.204.22@tcp2 [ 2360.556176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.676363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2360.702663] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2361.602632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2361.642727] LNet: Added LNI 192.168.204.22@tcp2 [8/256/0/180] [ 2362.725598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2362.777809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2362.836269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2363.086654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2363.119788] LNet: 117341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2363.127690] LNet: 117341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2363.133674] LNetError: 117341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2363.143844] LNetError: 117341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2363.150986] LNet: Removed LNI 192.168.204.22@tcp [ 2364.209133] LNet: Removed LNI 192.168.204.22@tcp2 [ 2364.265689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2364.327617] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2364.335278] LNet: Added LNI 192.168.204.22@tcp2 [8/256/0/180] [ 2364.342528] LNet: 117347:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2365.018945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2365.379618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.417106] LNet: Removed LNI 192.168.204.22@tcp [ 2366.502390] LNet: Removed LNI 192.168.204.22@tcp2 [ 2366.561147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2366.681970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2366.756851] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2367.646575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2367.680446] LNet: Added LNI 192.168.204.22@tcp2 [8/256/0/180] [ 2368.558424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2368.615212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2368.844663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2368.870814] LNetError: 117664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2368.882572] LNetError: 117664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2368.901785] LNet: Removed LNI 192.168.204.22@tcp [ 2368.910773] LNet: Removed LNI 192.168.204.22@tcp2 [ 2368.972416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2369.041494] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2369.048755] LNet: Added LNI 192.168.204.22@tcp2 [8/256/0/180] [ 2369.060866] LNet: 117670:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2369.768105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2370.092903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2371.167499] LNet: Removed LNI 192.168.204.22@tcp [ 2371.178732] LNet: Removed LNI 192.168.204.22@tcp2 [ 2371.210707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2371.265050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2371.303070] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2371.948680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2371.971724] LNet: Added LNI 192.168.204.22@tcp2 [8/256/0/180] [ 2372.983877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2373.053262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2373.130598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2373.436488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2373.456396] LNet: 117991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2373.461399] LNet: 117991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2374.495310] LNet: Removed LNI 192.168.204.22@tcp [ 2375.526238] LNet: Removed LNI 192.168.204.22@tcp2 [ 2375.555246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2375.629558] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2375.640527] LNet: Added LNI 192.168.204.22@tcp2 [8/256/0/180] [ 2375.645353] LNet: 117998:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2376.320783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2376.588965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2376.614768] LNet: Removed LNI 192.168.204.22@tcp [ 2377.639715] LNet: Removed LNI 192.168.204.22@tcp2 [ 2377.740519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2377.872223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2377.944216] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2378.897552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2378.924150] LNet: Added LNI 192.168.204.22@tcp2 [8/256/0/180] [ 2380.419965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2380.454157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2380.539781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2380.952726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2380.978903] LNetError: 118331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2380.984848] LNetError: 118331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2380.993990] LNet: Removed LNI 192.168.204.22@tcp [ 2381.016690] LNet: Removed LNI 192.168.204.22@tcp2 [ 2381.036770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2381.147461] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2381.157046] LNet: Added LNI 192.168.204.22@tcp2 [8/256/0/180] [ 2381.165748] LNet: 118337:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2381.937469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2382.328454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2383.391154] LNet: Removed LNI 192.168.204.22@tcp [ 2383.403570] LNet: Removed LNI 192.168.204.22@tcp2 [ 2383.446980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2383.511955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2383.554191] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2384.523045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2384.574419] LNet: Added LNI 192.168.204.22@tcp2 [8/256/0/180] [ 2386.072442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2386.141949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2386.194324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2386.252634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2386.595218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2387.679206] LNet: Removed LNI 192.168.204.22@tcp [ 2388.709599] LNet: Removed LNI 192.168.204.22@tcp2 [ 2388.737633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2388.836336] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2388.843337] LNet: Added LNI 192.168.204.22@tcp2 [8/256/0/180] [ 2389.740115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2398.313543] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2401.031428] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 18:38:57 (1790980737) [ 2401.170259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2401.216957] LNet: 119451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2401.227481] LNet: 119451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2401.269215] LNetError: 119451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2401.276055] LNetError: 119451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2401.290480] LNet: Removed LNI 192.168.204.22@tcp [ 2402.344632] LNet: Removed LNI 192.168.204.22@tcp2 [ 2402.401653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2402.836806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2402.866758] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2404.068956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2404.098471] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2405.195710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2405.231796] LNet: Removed LNI 10.1.2.3@tcp [ 2405.258468] LNet: Removed LNI 192.168.204.22@tcp [ 2405.284548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2405.391656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2405.442276] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2405.449394] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2406.067849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2406.109286] LNet: Removed LNI 192.168.204.22@tcp [ 2406.128758] LNet: Removed LNI 10.1.2.3@tcp [ 2413.031413] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 18:39:10 (1790980750) [ 2413.281699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2413.363807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2413.467986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2413.507090] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2414.331691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2414.384495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2414.456782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2414.498448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2414.533753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2414.566846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2414.596879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2414.637300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2414.667733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2414.705918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2414.736543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2414.785392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2414.950492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2422.235729] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 18:39:19 (1790980759) [ 2422.339714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2423.398580] LNet: Removed LNI 192.168.204.22@tcp [ 2423.441853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2423.541876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2423.579381] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2424.235542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2424.276496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2424.343914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2424.393858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2424.443106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2424.495794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2424.532774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2424.598851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2424.627896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2424.660954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2424.702445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2424.747384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2424.835471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2424.861506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2424.887754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2424.911934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2424.949190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2424.977773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2425.009069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2425.059698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2425.106426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2425.170950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2425.210567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.22@tcp [ 2433.002774] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 18:39:30 (1790980770) [ 2433.201137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2433.234355] LNet: 121827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2433.243859] LNet: 121827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2434.286387] LNet: Removed LNI 192.168.204.22@tcp [ 2434.375784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2434.593782] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2434.827894] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2434.830979] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2434.853175] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2437.158667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2437.193507] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2438.039587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2438.069613] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2440.079956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2440.123442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2440.198544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.22@tcp -m PUT -e local_timeout -r 1 [ 2440.255514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2440.307808] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2440.310553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2440.527232] LNetError: 121897:0:(lib-move.c:4794:lnet_parse()) 192.168.204.22@tcp, src 192.168.204.22@tcp: Dropping PUT (error -108 looking up sender) [ 2441.567942] LNetError: 122355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2441.574161] LNetError: 122355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2441.582557] LNet: Removed LNI 10.1.2.3@tcp [ 2442.603792] LNet: Removed LNI 192.168.204.22@tcp [ 2443.180521] Key type .llcrypt unregistered [ 2443.184246] Key type ._llcrypt unregistered [ 2460.347785] Key type ._llcrypt registered [ 2460.351127] Key type .llcrypt registered [ 2461.871657] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 18:39:59 (1790980799) [ 2462.532395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2462.559760] alg: No test for adler32 (adler32-zlib) [ 2463.525609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2464.026933] LNet: 123208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2464.804789] Key type .llcrypt unregistered [ 2464.810916] Key type ._llcrypt unregistered [ 2472.872922] Key type ._llcrypt registered [ 2472.878273] Key type .llcrypt registered [ 2473.187791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2473.220495] alg: No test for adler32 (adler32-zlib) [ 2474.143926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2474.171907] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2482.397870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 2482.655711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2483.509408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.122@tcp [ 2483.759290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2483.786229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2483.809683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2483.843327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2483.884497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2483.921275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2483.946363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2483.973134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2483.997847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2484.018363] LNet: 123660:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2484.036212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2484.080214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2484.117564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2484.149572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2484.194758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2484.234257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2484.755828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2485.001888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2485.027664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2485.059616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2485.091273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2485.127654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2485.164321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2485.206285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2485.255110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2485.292679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2485.330149] LNet: 123779:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2485.363976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2485.410388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2485.454421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2485.472544] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.122@tcp [ 2485.478336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2485.515444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2485.552790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2486.336273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.122@tcp [ 2486.667623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2486.738624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2486.790871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2486.845138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2486.884829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2486.940166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2486.997189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2487.053505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2487.089984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2487.112596] LNet: 123898:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2487.131982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2487.160441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2487.190442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2487.220416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2487.296536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2487.348096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2488.131916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2488.475184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2488.526486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2488.580169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2488.618950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2488.655467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2488.690962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2488.722672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2488.755573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2488.796212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2488.843834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2488.885660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2488.946851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2488.997368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2489.057874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2489.106994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2489.530318] LNet: 124064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2489.537467] LNetError: 124064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2489.555926] LNet: Removed LNI 192.168.204.22@tcp [ 2489.985567] Key type .llcrypt unregistered [ 2489.989346] Key type ._llcrypt unregistered [ 2510.052471] Key type ._llcrypt registered [ 2510.055239] Key type .llcrypt registered [ 2511.873171] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 18:40:48 (1790980848) [ 2512.582390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2512.612693] alg: No test for adler32 (adler32-zlib) [ 2513.558711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2514.074650] LNet: 124951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2514.780808] Key type .llcrypt unregistered [ 2514.783921] Key type ._llcrypt unregistered [ 2522.722187] Key type ._llcrypt registered [ 2522.725282] Key type .llcrypt registered [ 2523.023935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2523.056312] alg: No test for adler32 (adler32-zlib) [ 2524.030873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2524.068711] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2533.138622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 2542.834637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2542.891139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2542.941568] LNet: Added LNI 192.168.204.22@tcp1 [8/256/0/180] [ 2544.213237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.122@tcp [ 2544.455198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2545.249260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.122@tcp [ 2545.522097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2545.544562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2545.577336] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.122@tcp [ 2545.579233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2545.625718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2545.689611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2545.726437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2545.754275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2545.793758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2545.844194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2545.868435] LNet: 125612:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2545.892163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2545.929273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2545.979949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2546.031535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2546.067485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2546.119445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2546.856342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2547.111838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2547.135687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2547.171262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2547.204640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2547.250417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2547.292098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2547.336655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2547.365537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2547.413055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2547.446334] LNet: 125731:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2547.475802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2547.547828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2547.597433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2547.632540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2547.696971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2547.767624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2548.596384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.122@tcp [ 2548.908718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2548.948630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2548.978252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2549.006467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2549.035321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2549.066906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2549.102560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2549.144280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2549.180612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2549.212356] LNet: 125850:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2549.257360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2549.314598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2549.358201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2549.402078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2549.448122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2549.523790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2550.365399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2550.651848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2550.679364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2550.707961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2550.739776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2550.763715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2550.804173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2550.842655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2550.883609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2550.921969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2550.954733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2550.991429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2551.029730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2551.066556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2551.106399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2551.155906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2556.934843] LNet: 126051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2556.943826] LNetError: 126051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2556.956485] LNet: Removed LNI 192.168.204.22@tcp [ 2556.965876] LNet: Removed LNI 192.168.204.22@tcp1 [ 2557.385728] Key type .llcrypt unregistered [ 2557.387299] Key type ._llcrypt unregistered [ 2577.715377] Key type ._llcrypt registered [ 2577.718962] Key type .llcrypt registered [ 2579.395260] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 18:41:56 (1790980916) [ 2579.754735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2579.772802] alg: No test for adler32 (adler32-zlib) [ 2580.745422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2580.816692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2580.862455] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2581.644744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2581.663772] LNet: Added LNI 192.168.204.22@tcp2 [8/256/0/180] [ 2582.692441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.22@tcp [ 2582.821884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2583.903162] LNet: Removed LNI 192.168.204.22@tcp [ 2590.587526] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 18:42:08 (1790980928) [ 2592.805233] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2595.104092] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 18:42:12 (1790980932) [ 2602.214763] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 18:42:19 (1790980939) [ 2609.666917] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 18:42:27 (1790980947) [ 2617.090281] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 18:42:34 (1790980954) [ 2626.747715] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 18:42:43 (1790980963) [ 2639.064202] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 18:42:55 (1790980975) [ 2650.958385] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 18:43:07 (1790980987) [ 2659.497868] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 18:43:17 (1790980997) [ 2667.282776] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 18:43:24 (1790981004) [ 2675.868806] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 18:43:33 (1790981013) [ 2683.764527] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 18:43:41 (1790981021) [ 2692.116312] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 18:43:49 (1790981029) [ 2700.452972] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 18:43:57 (1790981037) [ 2708.199731] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 18:44:05 (1790981045) [ 2717.469236] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 18:44:14 (1790981054) [ 2725.337449] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 18:44:22 (1790981062) [ 2734.202401] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 18:44:31 (1790981071) [ 2741.385452] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 18:44:38 (1790981078) [ 2741.572553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2741.607938] LNet: 137754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2741.612855] LNetError: 137754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2741.632876] LNet: Removed LNI 192.168.204.22@tcp2 [ 2741.650587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2741.809499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2741.832407] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2743.193752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2743.231249] LNet: 137896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2743.238284] LNetError: 137896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2743.262860] LNet: Removed LNI 192.168.204.22@tcp [ 2743.310966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2743.390866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2743.437189] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2751.982986] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 18:44:49 (1790981089) [ 2761.371185] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 18:44:58 (1790981098) [ 2770.140482] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 18:45:07 (1790981107) [ 2779.690932] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 18:45:16 (1790981116) [ 2787.712904] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 18:45:25 (1790981125) [ 2795.408125] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 18:45:32 (1790981132) [ 2804.509900] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 18:45:41 (1790981141) [ 2813.500622] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 18:45:50 (1790981150) [ 2823.500605] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 18:46:00 (1790981160) [ 2831.859482] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 18:46:09 (1790981169) [ 2839.942862] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 18:46:17 (1790981177) [ 2848.492896] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 18:46:25 (1790981185) [ 2856.379694] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2602 sec ========== 18:46:33 (1790981193) [ 2856.980471] LNet: 145365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2856.990731] LNetError: 145365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2857.005115] LNet: Removed LNI 192.168.204.22@tcp [ 2857.593185] Key type .llcrypt unregistered [ 2857.596405] Key type ._llcrypt unregistered [ 2870.801554] Key type ._llcrypt registered [ 2870.813475] Key type .llcrypt registered [ 2871.521770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2871.576569] alg: No test for adler32 (adler32-zlib) [ 2872.753047] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 2873.088311] LNet: Added LNI 192.168.204.22@tcp [8/256/0/180] [ 2874.815670] Key type lgssc registered [ 2876.469288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2947.318953] Lustre: Mounted lustre-client - version 2.17.59_2_ge06ed12 [ 2953.174761] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2962.603369] Lustre: Unmounted lustre-client [ 2988.397506] Key type lgssc unregistered [ 2988.690809] LNet: 148327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2988.704781] LNetError: 148327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2988.742911] LNet: Removed LNI 192.168.204.22@tcp [ 2989.725185] Key type .llcrypt unregistered [ 2989.740617] Key type ._llcrypt unregistered