[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 440253328 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002324] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.006660] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009141] LSM: Security Framework initializing [ 0.010059] Yama: becoming mindful. [ 0.011044] SELinux: Initializing. [ 0.012059] *** VALIDATE selinux *** [ 0.020863] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025652] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026228] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027140] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028139] *** VALIDATE tmpfs *** [ 0.030165] *** VALIDATE proc *** [ 0.031299] *** VALIDATE cgroup *** [ 0.032014] *** VALIDATE cgroup2 *** [ 0.033280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037032] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041361] debug: unmapping init [mem 0xffffffff89e59000-0xffffffff89e60fff] [ 0.044214] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045723] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046030] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053319] rcu: Hierarchical SRCU implementation. [ 0.055523] smp: Bringing up secondary CPUs ... [ 0.056629] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.060324] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142024] node 0 deferred pages initialised in 76ms [ 0.146010] devtmpfs: initialized [ 0.147257] x86/mm: Memory block size: 128MB [ 0.149846] gcov: version magic: 0x41383552 [ 0.152317] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154282] pinctrl core: initialized pinctrl subsystem [ 0.155240] [ 0.155813] ************************************************************* [ 0.156013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157013] ** ** [ 0.158012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159013] ** ** [ 0.160016] ** This means that this kernel is built to expose internal ** [ 0.161014] ** IOMMU data structures, which may compromise security on ** [ 0.162011] ** your system. ** [ 0.163013] ** ** [ 0.164023] ** If you see this message and you are not debugging the ** [ 0.165013] ** kernel, report this immediately to your vendor! ** [ 0.166014] ** ** [ 0.167016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168013] ************************************************************* [ 0.169655] NET: Registered protocol family 16 [ 0.170466] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.171059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173467] cpuidle: using governor menu [ 0.174000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176251] PCI: Using configuration type 1 for base access [ 0.178147] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186115] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.187024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.188182] cryptd: max_cpu_qlen set to 1000 [ 0.190048] ACPI: Added _OSI(Module Device) [ 0.191014] ACPI: Added _OSI(Processor Device) [ 0.192016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.193014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.196922] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.198577] ACPI: Interpreter enabled [ 0.199062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.200014] ACPI: Using IOAPIC for interrupt routing [ 0.201133] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.202390] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.212351] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.213052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.214022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.215089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.217327] acpiphp: Slot [2] registered [ 0.218279] acpiphp: Slot [5] registered [ 0.219130] acpiphp: Slot [6] registered [ 0.220113] acpiphp: Slot [3] registered [ 0.221125] acpiphp: Slot [4] registered [ 0.222105] acpiphp: Slot [7] registered [ 0.223129] acpiphp: Slot [8] registered [ 0.224107] acpiphp: Slot [9] registered [ 0.225108] acpiphp: Slot [10] registered [ 0.226094] acpiphp: Slot [11] registered [ 0.227086] acpiphp: Slot [12] registered [ 0.228093] acpiphp: Slot [13] registered [ 0.229081] acpiphp: Slot [14] registered [ 0.230071] acpiphp: Slot [15] registered [ 0.231112] acpiphp: Slot [16] registered [ 0.232083] acpiphp: Slot [17] registered [ 0.233075] acpiphp: Slot [18] registered [ 0.234084] acpiphp: Slot [19] registered [ 0.235105] acpiphp: Slot [20] registered [ 0.236088] acpiphp: Slot [21] registered [ 0.237101] acpiphp: Slot [22] registered [ 0.238083] acpiphp: Slot [23] registered [ 0.239121] acpiphp: Slot [24] registered [ 0.240089] acpiphp: Slot [25] registered [ 0.241099] acpiphp: Slot [26] registered [ 0.242094] acpiphp: Slot [27] registered [ 0.243077] acpiphp: Slot [28] registered [ 0.244050] acpiphp: Slot [29] registered [ 0.245023] acpiphp: Slot [30] registered [ 0.245801] acpiphp: Slot [31] registered [ 0.246054] PCI host bridge to bus 0000:00 [ 0.247012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.248017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.249016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.250017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.251014] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.252019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.253172] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.254699] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.255819] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.259409] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.262051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.263021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.264022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.265019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.266454] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.269851] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.272045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.275819] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.280013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.289722] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.294013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.300000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.306018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.313014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.332020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.339000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.347019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.361020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.378019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.389092] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.391386] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.393381] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.396451] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.398212] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.402179] iommu: Default domain type: Passthrough [ 0.404584] SCSI subsystem initialized [ 0.406094] ACPI: bus type USB registered [ 0.407089] usbcore: registered new interface driver usbfs [ 0.408072] usbcore: registered new interface driver hub [ 0.409081] usbcore: registered new device driver usb [ 0.411136] pps_core: LinuxPPS API ver. 1 registered [ 0.412013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.414056] PTP clock support registered [ 0.416084] EDAC MC: Ver: 3.0.0 [ 0.417139] PCI: Using ACPI for IRQ routing [ 0.418754] NetLabel: Initializing [ 0.419009] NetLabel: domain hash size = 128 [ 0.420023] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.421090] NetLabel: unlabeled traffic allowed by default [ 0.422112] vgaarb: loaded [ 0.423252] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.425012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.430053] clocksource: Switched to clocksource kvm-clock [ 0.514656] VFS: Disk quotas dquot_6.6.0 [ 0.516608] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.518942] *** VALIDATE ramfs *** [ 0.519874] *** VALIDATE hugetlbfs *** [ 0.522197] pnp: PnP ACPI init [ 0.524703] pnp: PnP ACPI: found 6 devices [ 0.543721] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.547085] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.549211] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.551285] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.553403] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.555749] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.558541] NET: Registered protocol family 2 [ 0.561438] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.565109] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.567634] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.571682] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.574358] TCP: Hash tables configured (established 65536 bind 65536) [ 0.576443] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.578583] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.580658] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.582768] NET: Registered protocol family 1 [ 0.585203] RPC: Registered named UNIX socket transport module. [ 0.586697] RPC: Registered udp transport module. [ 0.587742] RPC: Registered tcp transport module. [ 0.588861] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.590744] NET: Registered protocol family 44 [ 0.592407] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.594566] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.596699] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.599121] PCI: CLS 0 bytes, default 64 [ 0.601440] Unpacking initramfs... [ 1.969470] debug: unmapping init [mem 0xffff8ef4fcc64000-0xffff8ef4fffcffff] [ 1.973318] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.975460] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.978218] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.466467] Initialise system trusted keyrings [ 2.468232] Key type blacklist registered [ 2.470097] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.483196] zbud: loaded [ 2.488155] *** VALIDATE nfs *** [ 2.489344] *** VALIDATE nfs4 *** [ 2.490834] pstore: using deflate compression [ 2.495612] Platform Keyring initialized [ 2.617868] NET: Registered protocol family 38 [ 2.619658] Key type asymmetric registered [ 2.621293] Asymmetric key parser 'x509' registered [ 2.623391] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.626765] io scheduler mq-deadline registered [ 2.628991] io scheduler kyber registered [ 2.630936] io scheduler bfq registered [ 2.635054] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.637850] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.641109] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.643450] ACPI: Power Button [PWRF] [ 2.648180] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.655496] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.676504] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.703844] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.732159] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.738197] Non-volatile memory driver v1.3 [ 2.740035] Linux agpgart interface v0.103 [ 2.783185] virtio_blk virtio1: [vda] 145736 512-byte logical blocks (74.6 MB/71.2 MiB) [ 2.785888] vda: detected capacity change from 0 to 74616832 [ 2.800783] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.803910] vdb: detected capacity change from 0 to 1073741824 [ 2.815199] libphy: Fixed MDIO Bus: probed [ 2.822064] usbcore: registered new interface driver usbserial_generic [ 2.826098] usbserial: USB Serial support registered for generic [ 2.828945] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.834219] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.836557] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.839581] mousedev: PS/2 mouse device common for all mice [ 2.842413] rtc_cmos 00:05: RTC can wake from S4 [ 2.845494] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.845564] rtc_cmos 00:05: registered as rtc0 [ 2.850912] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.852294] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.854340] intel_pstate: CPU model not supported [ 2.859308] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.863257] hid: raw HID events driver (C) Jiri Kosina [ 2.865282] usbcore: registered new interface driver usbhid [ 2.867265] usbhid: USB HID core driver [ 2.868877] drop_monitor: Initializing network drop monitor service [ 2.871361] Initializing XFRM netlink socket [ 2.873324] NET: Registered protocol family 10 [ 2.876470] Segment Routing with IPv6 [ 2.878035] NET: Registered protocol family 17 [ 2.879727] mpls_gso: MPLS GSO support [ 2.886840] RAS: Correctable Errors collector initialized. [ 2.889018] AVX version of gcm_enc/dec engaged. [ 2.890406] AES CTR mode by8 optimization enabled [ 2.968515] sched_clock: Marking stable (2968482683, 0)->(3908846097, -940363414) [ 2.972467] registered taskstats version 1 [ 2.975147] Loading compiled-in X.509 certificates [ 2.977178] zswap: loaded using pool lzo/zbud [ 3.001192] Key type big_key registered [ 3.012453] Key type encrypted registered [ 3.014176] ima: No TPM chip found, activating TPM-bypass! [ 3.016255] ima: Allocated hash algorithm: sha1 [ 3.017734] ima: No architecture policies found [ 3.020422] evm: Initialising EVM extended attributes: [ 3.021903] evm: security.selinux [ 3.023124] evm: security.ima [ 3.024188] evm: security.capability [ 3.025480] evm: HMAC attrs: 0x1 [ 3.027881] rtc_cmos 00:05: setting system clock to 2026-07-13 13:54:37 UTC (1783950877) [ 3.033715] debug: unmapping init [mem 0xffffffff8ae03000-0xffffffff8affffff] [ 3.036642] debug: unmapping init [mem 0xffffffff89b82000-0xffffffff89e58fff] [ 3.046070] Write protecting the kernel read-only data: 28672k [ 3.048880] debug: unmapping init [mem 0xffffffff88203000-0xffffffff883fffff] [ 3.050825] debug: unmapping init [mem 0xffffffff88b14000-0xffffffff88bfffff] [ 3.080562] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.089729] systemd[1]: Detected virtualization kvm. [ 3.091641] systemd[1]: Detected architecture x86-64. [ 3.093530] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.117674] systemd[1]: No hostname configured. [ 3.119509] systemd[1]: Set hostname to . [ 3.121548] random: systemd: uninitialized urandom read (16 bytes read) [ 3.123918] systemd[1]: Initializing machine ID from random generator. [ 3.163387] random: ln: uninitialized urandom read (6 bytes read) [ 3.248761] random: systemd: uninitialized urandom read (16 bytes read) [ 3.251750] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.257125] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.262554] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.824284] device-mapper: uevent: version 1.0.3 [ 3.826789] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.429042] virtio_net virtio0 ens2: renamed from eth0 [ 4.441410] random: fast init done [ 4.456217] scsi host0: ata_piix [ 4.461765] scsi host1: ata_piix [ 4.463479] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.465736] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.118207] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.532989] random: crng init done [ 9.534349] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.801683] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.953633] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.187659] SELinux: Disabled at runtime. [ 11.245524] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.252471] systemd[1]: Detected virtualization kvm. [ 11.254162] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.670558] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.672876] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.677067] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.681263] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.684266] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.690438] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.695653] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 11.746581] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.105614] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.398349] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.430466] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.510300] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.523805] EDAC sbridge: Ver: 1.1.2 [ 13.568755] Key type dns_resolver registered [ 13.867975] NFS: Registering the id_resolver key type [ 13.870093] Key type id_resolver registered [ 13.871961] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg153-client login: [ 44.817989] libcfs: loading out-of-tree module taints kernel. [ 44.911226] Key type ._llcrypt registered [ 44.912726] Key type .llcrypt registered [ 45.143313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 45.148899] alg: No test for adler32 (adler32-zlib) [ 46.157472] Lustre: Lustre: Build Version: 2.17.54_162_gb5ded84 [ 46.480507] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 48.111194] Key type lgssc registered [ 48.741804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 103.821957] hrtimer: interrupt took 17947850 ns [ 184.602451] Lustre: Mounted lustre-client [ 190.125159] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 205.065982] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing check_logdir /tmp/testlogs/ [ 210.331924] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing yml_node [ 210.400536] Lustre: lustre-OST0000-osc-ffff8ef5501d7000: disconnect after 23s idle [ 214.115434] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 216.507666] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 219.304939] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 220.878772] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Jul 13 09:58:13 EDT 2026 [ 237.746825] Lustre: Unmounted lustre-client [ 256.415419] Key type lgssc unregistered [ 256.646546] LNet: 8389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 256.653799] LNetError: 8389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 256.698399] LNet: Removed LNI 192.168.201.53@tcp [ 257.416351] Key type .llcrypt unregistered [ 257.419937] Key type ._llcrypt unregistered [ 282.046528] Key type ._llcrypt registered [ 282.052415] Key type .llcrypt registered [ 282.211433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 282.228504] alg: No test for adler32 (adler32-zlib) [ 283.162530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 283.220614] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 283.244481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 287.709602] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 287.923279] LNet: 9271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 287.938499] LNetError: 9271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 287.984744] LNet: Removed LNI 192.168.201.53@tcp [ 288.676803] Key type .llcrypt unregistered [ 288.679210] Key type ._llcrypt unregistered [ 294.412361] Key type ._llcrypt registered [ 294.415478] Key type .llcrypt registered [ 295.772064] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:59:28 (1783951168) [ 296.165470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 296.174644] alg: No test for adler32 (adler32-zlib) [ 297.068534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.135206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.197049] LNet: 9684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 297.230517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.301286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 297.330470] LNet: 9691:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 303.402380] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:59:36 (1783951176) [ 303.689225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.735515] LNet: 10268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 303.793166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.998568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 310.364981] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:59:43 (1783951183) [ 310.639288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.673408] LNet: 10853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.703358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.761701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 318.588483] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:59:51 (1783951191) [ 318.799025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.831144] LNet: 11435:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.857417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.925965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 326.366968] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:59:58 (1783951198) [ 326.538359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 326.612723] LNet: 12017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.655319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.741660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 327.434853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 335.870639] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:00:08 (1783951208) [ 336.038441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.073650] LNet: 12605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.121984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.194595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 336.507751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 336.737890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 342.939562] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:00:15 (1783951215) [ 343.067540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.113892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.209931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 350.887433] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:00:23 (1783951223) [ 350.959151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.980103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.015920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 351.031118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 351.389503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 351.428974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 351.932265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 352.002922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 352.547351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 352.599379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 352.901382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 352.927783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 353.173397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 353.203868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 353.521125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 353.552425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 353.811592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 353.833219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 354.302850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 354.335807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 354.627787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 354.651888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 354.905283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 354.937377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 355.340736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 355.357347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 355.689338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 355.731348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 362.639641] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:00:35 (1783951235) [ 362.946964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.977841] LNet: 14480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.989505] LNet: 14480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 363.032200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.112969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 363.141939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 369.851952] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:00:42 (1783951242) [ 370.081831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.166758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.262881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 370.296346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 376.641427] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:00:49 (1783951249) [ 376.770217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.836164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.929501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 376.960437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 384.232104] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:00:57 (1783951257) [ 384.789216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.868628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.939571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 384.969476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 393.118700] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:01:06 (1783951266) [ 393.203666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.242166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.327648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 393.360853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 400.488478] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:01:13 (1783951273) [ 400.780636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.808212] LNet: 17425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.822693] LNet: 17425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 400.878623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.005508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 401.041725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 408.021723] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:01:21 (1783951281) [ 408.272358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.332775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.407668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 408.697351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 415.360990] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:01:28 (1783951288) [ 415.765424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.824547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.247602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 416.569611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 424.084713] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:01:37 (1783951297) [ 424.163635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.202629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.361398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 424.555361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 431.388499] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:01:44 (1783951304) [ 431.863565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.928505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.871736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 433.268184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 440.896208] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:01:54 (1783951314) [ 441.744974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.837858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.962562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 442.012967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 450.135952] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:02:03 (1783951323) [ 450.682789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.705583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.760457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 450.811880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 458.754701] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:02:11 (1783951331) [ 458.904238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.009942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.117918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 468.469595] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:02:21 (1783951341) [ 468.984622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.051946] LNet: 22279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.062360] LNet: 22279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 469.163585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.393808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 477.482511] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:02:30 (1783951350) [ 477.735638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.764524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.839390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 478.476663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 486.085880] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:02:39 (1783951359) [ 486.338185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.364491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.475982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 486.521346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 493.762673] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:02:46 (1783951366) [ 493.875732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.951522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.073767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 494.135664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 500.983102] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:02:54 (1783951374) [ 501.130083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.228420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.316727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 501.376387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 508.483403] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:03:01 (1783951381) [ 508.566673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.634876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.732396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 508.773186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 514.783296] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:03:08 (1783951388) [ 515.217616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.272407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.322490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 515.385419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 515.406485] LNetError: 25812:0:(peer.c:2276:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 515.456392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 515.483795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 522.071581] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:03:15 (1783951395) [ 522.310273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.371917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.465960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.5oBIFD8N2x [ 522.599286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.5oBIFD8N2x [ 522.709636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.5oBIFD8N2x [ 529.403654] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:03:22 (1783951402) [ 529.480080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.556474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.687441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 529.766631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 529.836189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 536.529317] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 10:03:29 (1783951409) [ 536.898538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.935895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.995661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 537.017497] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 538.752615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 540.176921] Lustre: Lustre: Build Version: 2.17.54_162_gb5ded84 [ 542.095180] Key type lgssc registered [ 543.180206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 546.248750] LNetError: 27574:0:(lib-move.c:2120:lnet_handle_find_routed_path()) peer 192.168.201.53@tcp99 has no available nets [ 548.296099] Key type lgssc unregistered [ 548.549837] LNetError: 28682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 548.562738] LNet: Removed LNI 192.168.201.53@tcp [ 548.914456] Key type .llcrypt unregistered [ 548.915349] Key type ._llcrypt unregistered [ 559.529868] Key type ._llcrypt registered [ 559.531905] Key type .llcrypt registered [ 561.524513] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 10:03:53 (1783951433) [ 562.307193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 562.334967] alg: No test for adler32 (adler32-zlib) [ 563.343262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.419226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 565.804656] LNet: Message forwarding will be enabled in 2s [ 568.864893] LNet: Message forwarding enabled [ 569.527498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 569.552897] LNet: Message forwarding will stop in 2s to 6s [ 573.139523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 574.455411] LNet: Message forwarding disabled [ 575.641170] LNet: Message forwarding will be enabled in 2s [ 579.616154] LNet: Message forwarding enabled [ 584.788610] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 10:04:18 (1783951458) [ 585.115109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.159569] LNet: 30083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 586.733927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.810664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 592.511614] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:04:25 (1783951465) [ 592.696318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.721034] LNet: 30664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.744512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.802973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 592.830499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 592.853432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 592.880152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 592.909454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 592.942206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 592.966077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 592.980330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 593.003792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 593.034292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 593.073544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 593.098410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 593.125745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 593.151605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 593.173755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 593.210550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 593.234340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 593.255189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 593.271097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 593.280546] LNetError: 30707:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 599.712825] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:04:32 (1783951472) [ 599.843283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.886896] LNet: 31276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 599.933894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.019416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 606.121513] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:04:39 (1783951479) [ 608.279733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg153-server [ 610.071239] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:04:43 (1783951483) [ 610.379432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.404549] LNet: 32208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.433398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.497313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 610.521948] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 611.400100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[54-58]@tcp [ 611.425678] LNet: 32342:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 611.496796] LNetError: 32212:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.55@tcp is being used as a gateway but routing feature is not turned on [ 611.532389] LNetError: 32212:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.55@tcp has gone from up to down [ 612.532517] LNetError: 32212:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.54@tcp is being used as a gateway but routing feature is not turned on [ 612.542685] LNetError: 32212:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 613.548907] LNetError: 32212:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.56@tcp is being used as a gateway but routing feature is not turned on [ 613.557476] LNetError: 32212:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 615.591485] LNetError: 32212:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.55@tcp is being used as a gateway but routing feature is not turned on [ 615.616102] LNetError: 32212:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 618.535660] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:04:51 (1783951491) [ 619.712863] LNetError: 32212:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.54@tcp is being used as a gateway but routing feature is not turned on [ 619.734551] LNetError: 32212:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 620.766355] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg153-server [ 622.423752] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:04:55 (1783951495) [ 622.606176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.621123] LNet: 33276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 622.625494] LNetError: 33276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 623.668836] LNet: Removed LNI 192.168.201.53@tcp [ 623.718486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.793673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 623.810597] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 624.583105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[54-59/2]@tcp [ 624.611563] LNet: 33395:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 624.618593] LNet: 33395:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 624.653166] LNetError: 33282:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.56@tcp has gone from up to down [ 624.658305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[54-59/2]@tcp [ 624.660902] LNetError: 33282:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 631.981517] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:05:05 (1783951505) [ 632.206803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.233326] LNet: 33987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.242744] LNetError: 33987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 632.275183] LNet: Removed LNI 192.168.201.53@tcp [ 632.296632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.386878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 632.452747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.491919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.555728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 632.604304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.633932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.683541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 632.766731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.802398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.884733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 632.997903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.043724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.102937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 633.192834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.225658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.293939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 633.439135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.480329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.546471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 633.692720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.737528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.791557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 633.863983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.909221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.978965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 634.088436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.126621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.188493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 634.288939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.341563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.408846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 634.472348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.516418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.580484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 640.935781] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:05:14 (1783951514) [ 642.842935] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg153-server [ 644.349876] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:05:17 (1783951517) [ 645.960493] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg153-server [ 647.683096] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:05:20 (1783951520) [ 648.315741] Key type .llcrypt unregistered [ 648.318494] Key type ._llcrypt unregistered [ 654.414888] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 654.615269] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 654.638435] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 654.653996] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 657.358682] Key type ._llcrypt registered [ 657.360498] Key type .llcrypt registered [ 657.595045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 657.620308] alg: No test for adler32 (adler32-zlib) [ 658.556488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.650228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 658.676635] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 659.375627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 659.395812] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 661.233715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 661.267356] LNet: Removed LNI 10.1.2.3@tcp [ 661.430555] LNet: 36105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.447161] LNetError: 36105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 661.470953] LNet: Removed LNI 192.168.201.53@tcp [ 662.216443] Key type .llcrypt unregistered [ 662.217815] Key type ._llcrypt unregistered [ 673.158240] Key type ._llcrypt registered [ 673.164378] Key type .llcrypt registered [ 674.386646] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:05:47 (1783951547) [ 674.765463] Key type .llcrypt unregistered [ 674.766888] Key type ._llcrypt unregistered [ 680.596295] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 680.799300] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 680.812857] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 680.815217] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 683.639715] Key type ._llcrypt registered [ 683.641209] Key type .llcrypt registered [ 683.831521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 683.847593] alg: No test for adler32 (adler32-zlib) [ 684.754167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.833948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 684.873762] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 685.463770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 685.486282] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 687.842126] LNet: 37598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.856918] LNetError: 37598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 687.868861] LNet: Removed LNI 10.1.2.3@tcp [ 687.887749] LNet: Removed LNI 192.168.201.53@tcp [ 688.428679] Key type .llcrypt unregistered [ 688.430788] Key type ._llcrypt unregistered [ 699.166339] Key type ._llcrypt registered [ 699.167872] Key type .llcrypt registered [ 700.051181] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:06:13 (1783951573) [ 700.599916] Key type .llcrypt unregistered [ 700.602259] Key type ._llcrypt unregistered [ 705.643921] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 705.706420] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 705.713414] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 705.748268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 708.440656] Key type ._llcrypt registered [ 708.442593] Key type .llcrypt registered [ 708.574839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 708.583723] alg: No test for adler32 (adler32-zlib) [ 709.504287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.543346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 709.564964] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 711.238091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 711.260737] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 713.176688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 713.194919] LNet: Removed LNI 10.1.2.3@tcp [ 713.213403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 713.254782] LNet: Removed LNI 10.1.2.31@tcp [ 713.271708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 713.295594] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 713.936580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 713.952607] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 714.712310] LNet: 39518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 714.717474] LNetError: 39518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 714.721645] LNet: Removed LNI 10.1.2.31@tcp [ 714.727547] LNet: Removed LNI 10.1.2.3@tcp [ 715.187439] Key type .llcrypt unregistered [ 715.188581] Key type ._llcrypt unregistered [ 725.181463] Key type ._llcrypt registered [ 725.183339] Key type .llcrypt registered [ 726.339534] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 10:06:39 (1783951599) [ 726.789812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 726.807146] alg: No test for adler32 (adler32-zlib) [ 727.741683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.796252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 727.824267] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 728.645899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.675109] LNet: 40470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.680743] LNetError: 40470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.692781] LNet: Removed LNI 192.168.201.53@tcp [ 728.710823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.754233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.53@tcp [ 728.772916] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 729.572422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.53@tcp [ 729.601771] LNet: Removed LNI 192.168.201.53@tcp [ 729.639297] LNet: 40605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.099484] Key type .llcrypt unregistered [ 730.103147] Key type ._llcrypt unregistered [ 741.142871] Key type ._llcrypt registered [ 741.144812] Key type .llcrypt registered [ 742.608940] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 10:06:55 (1783951615) [ 744.562462] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg153-server [ 746.287789] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 10:06:59 (1783951619) [ 746.697443] Key type .llcrypt unregistered [ 746.701070] Key type ._llcrypt unregistered [ 752.318964] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 752.510041] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 752.521301] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 752.533441] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 755.116218] Key type ._llcrypt registered [ 755.131031] Key type .llcrypt registered [ 755.342389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 755.356681] alg: No test for adler32 (adler32-zlib) [ 756.345953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.456178] LNet: Added LNI 192.168.201.53@tcp1 [8/256/0/180] [ 757.304468] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 759.574437] LNet: 42444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.581593] LNetError: 42444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 759.587672] LNet: Removed LNI 192.168.201.53@tcp1 [ 759.599870] LNet: Removed LNI 10.1.2.3@tcp2 [ 760.346456] Key type .llcrypt unregistered [ 760.348110] Key type ._llcrypt unregistered [ 771.083748] Key type ._llcrypt registered [ 771.087267] Key type .llcrypt registered [ 772.873680] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 10:07:25 (1783951645) [ 773.344441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 773.367858] alg: No test for adler32 (adler32-zlib) [ 774.342463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.400745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 774.425063] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 775.177404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.214818] LNet: 43415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.221389] LNetError: 43415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 775.241257] LNet: Removed LNI 192.168.201.53@tcp [ 775.270595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.356369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 775.392241] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 782.913177] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 10:07:36 (1783951656) [ 783.173887] LNet: 44006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.179869] LNetError: 44006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.216513] LNet: Removed LNI 192.168.201.53@tcp [ 783.881733] Key type .llcrypt unregistered [ 783.883664] Key type ._llcrypt unregistered [ 792.095719] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 792.346817] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 792.363202] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 792.417851] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 795.528827] Key type ._llcrypt registered [ 795.536680] Key type .llcrypt registered [ 795.881719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 795.913837] alg: No test for adler32 (adler32-zlib) [ 796.825468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.893619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 796.943657] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 797.903193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 797.928928] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.818604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.843769] LNet: 44749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.851409] LNetError: 44749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 800.854284] LNet: Removed LNI 10.1.2.3@tcp [ 800.863926] LNet: Removed LNI 192.168.201.53@tcp [ 800.882063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.941666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 800.982345] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 800.985477] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 801.342570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.378549] LNet: 44776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.386823] LNetError: 44776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 801.390597] LNetError: 44776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 801.399331] LNet: Removed LNI 10.1.2.3@tcp [ 801.429143] LNet: Removed LNI 192.168.201.53@tcp [ 801.468189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.560492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 801.614716] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 801.621801] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 802.271115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 802.310713] LNet: Added LNI 192.168.201.53@tcp2 [8/256/0/180] [ 803.869277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.902039] LNet: 44939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.912330] LNetError: 44939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 803.917573] LNetError: 44939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 803.926972] LNet: Removed LNI 10.1.2.3@tcp [ 803.931520] LNet: Removed LNI 192.168.201.53@tcp [ 803.939965] LNet: Removed LNI 192.168.201.53@tcp2 [ 803.963687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.134121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 804.203565] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 804.220330] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 804.234437] LNet: Added LNI 192.168.201.53@tcp2 [8/256/0/180] [ 813.099431] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 10:08:05 (1783951685) [ 813.231569] LNet: 45526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.238367] LNetError: 45526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 813.243802] LNetError: 45526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 813.254562] LNet: Removed LNI 10.1.2.3@tcp [ 813.263393] LNet: Removed LNI 192.168.201.53@tcp [ 813.281683] LNet: Removed LNI 192.168.201.53@tcp2 [ 814.161667] Key type .llcrypt unregistered [ 814.164056] Key type ._llcrypt unregistered [ 819.895888] Key type ._llcrypt registered [ 819.897617] Key type .llcrypt registered [ 820.116764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 820.151368] alg: No test for adler32 (adler32-zlib) [ 821.212791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.325085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 821.406176] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 822.333145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 822.342460] LNet: 45929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.344977] LNetError: 45929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 822.354551] LNet: Removed LNI 192.168.201.53@tcp [ 822.372337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.490215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 822.524594] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 823.042884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 823.065330] LNet: 45959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.071628] LNetError: 45959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.087864] LNet: Removed LNI 192.168.201.53@tcp [ 823.106120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.169643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 823.200372] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 823.559226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 823.607180] LNet: Removed LNI 192.168.201.53@tcp [ 823.627041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.717740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 823.762213] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 824.098901] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 824.276711] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 824.280705] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 824.284801] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 826.769373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 826.823783] LNet: 46069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.828430] LNet: 46069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 826.832732] LNetError: 46069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.836502] LNetError: 46069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 826.847186] LNet: Removed LNI 192.168.201.53@tcp [ 826.909726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.987192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 827.012845] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 835.498409] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 10:08:28 (1783951708) [ 835.631500] LNet: 46658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.635897] LNetError: 46658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.682351] LNet: Removed LNI 192.168.201.53@tcp [ 836.650359] Key type .llcrypt unregistered [ 836.653112] Key type ._llcrypt unregistered [ 843.780441] Key type ._llcrypt registered [ 843.785554] Key type .llcrypt registered [ 844.131087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 844.163987] alg: No test for adler32 (adler32-zlib) [ 845.036424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.103485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 845.133121] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 846.161813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.180640] LNet: 47064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.187265] LNetError: 47064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.201393] LNet: Removed LNI 192.168.201.53@tcp [ 846.223862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.271533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 846.297234] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 852.044886] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 10:08:45 (1783951725) [ 852.301557] LNet: 47656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.320497] LNetError: 47656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.339911] LNet: Removed LNI 192.168.201.53@tcp [ 852.896610] Key type .llcrypt unregistered [ 852.898688] Key type ._llcrypt unregistered [ 858.604661] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 858.774543] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 858.782489] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 858.789988] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 861.415164] Key type ._llcrypt registered [ 861.422277] Key type .llcrypt registered [ 861.686193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 861.697230] alg: No test for adler32 (adler32-zlib) [ 862.651874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.696457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 862.715884] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 863.669906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.688488] LNet: 48100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.693787] LNetError: 48100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.707363] LNet: Removed LNI 192.168.201.53@tcp [ 863.726655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.803395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 863.823919] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 870.503444] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 10:09:03 (1783951743) [ 870.757635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.780843] LNet: 48698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.788708] LNetError: 48698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.804818] LNet: Removed LNI 192.168.201.53@tcp [ 870.824160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.937362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 876.257320] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 10:09:09 (1783951749) [ 876.365255] LNet: 49273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.163612] Key type .llcrypt unregistered [ 877.165704] Key type ._llcrypt unregistered [ 882.614335] Key type ._llcrypt registered [ 882.616973] Key type .llcrypt registered [ 882.830532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 882.860699] alg: No test for adler32 (adler32-zlib) [ 883.814774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.910042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 883.939478] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 884.902185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.917923] LNet: 49677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.923533] LNetError: 49677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.933447] LNet: Removed LNI 192.168.201.53@tcp [ 884.949722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.017145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 885.047126] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 891.294426] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 10:09:24 (1783951764) [ 891.621893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.676938] LNet: 50270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.685889] LNetError: 50270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.725599] LNet: Removed LNI 192.168.201.53@tcp [ 891.760168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.846181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 898.288304] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 10:09:31 (1783951771) [ 900.018318] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 902.144299] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 10:09:35 (1783951775) [ 903.721858] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 905.476392] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 10:09:38 (1783951778) [ 906.939625] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 908.791575] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 10:09:41 (1783951781) [ 910.277876] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 911.713190] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 10:09:45 (1783951785) [ 913.670624] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 915.379934] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 10:09:48 (1783951788) [ 915.705892] LNet: 52324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.581823] Key type .llcrypt unregistered [ 916.586117] Key type ._llcrypt unregistered [ 923.215836] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 923.311402] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 923.314783] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 923.318234] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 924.789925] Key type ._llcrypt registered [ 924.792901] Key type .llcrypt registered [ 924.967453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 924.980996] alg: No test for adler32 (adler32-zlib) [ 926.009353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.096838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 926.144741] LNet: Added LNI 192.168.201.53@tcp [8/512/0/180] [ 926.153346] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 926.157092] LNet: 52635:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 926.256551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.288651] LNet: 52654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.292335] LNetError: 52654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 926.301988] LNet: Removed LNI 10.1.2.3@tcp [ 926.304976] LNetError: 52654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.322132] LNet: Removed LNI 192.168.201.53@tcp [ 926.342352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.463261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 926.488550] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 926.574759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.604285] LNet: 52678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.614793] LNetError: 52678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.633313] LNet: Removed LNI 192.168.201.53@tcp [ 926.665994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.736260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 926.765277] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 926.772538] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 926.776559] LNet: 52687:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 926.874601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.892435] LNet: 52705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.898426] LNetError: 52705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 926.903748] LNet: Removed LNI 10.1.2.3@tcp [ 926.906821] LNetError: 52705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.918390] LNet: Removed LNI 192.168.201.53@tcp [ 926.931545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.991271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 927.010909] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 927.015768] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 927.017515] LNet: 52713:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 927.168295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.190055] LNet: 52739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.194853] LNetError: 52739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 927.198825] LNet: Removed LNI 10.1.2.3@tcp [ 927.200871] LNetError: 52739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.212476] LNet: Removed LNI 192.168.201.53@tcp [ 927.224340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.278711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 927.305732] LNet: Added LNI 192.168.201.53@tcp [8/256/0/182] [ 927.309790] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 927.411509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.430955] LNet: 52767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.436621] LNetError: 52767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 927.441695] LNet: Removed LNI 10.1.2.3@tcp [ 927.443984] LNetError: 52767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.454666] LNet: Removed LNI 192.168.201.53@tcp [ 927.483528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.543844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 927.568981] LNet: Added LNI 192.168.201.53@tcp [8/256/0/182] [ 927.576518] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 927.579231] LNet: 52775:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 927.657785] LNet: Removed LNI 10.1.2.3@tcp [ 927.671627] LNet: Removed LNI 192.168.201.53@tcp [ 928.131881] Key type .llcrypt unregistered [ 928.135694] Key type ._llcrypt unregistered [ 938.166610] Key type ._llcrypt registered [ 938.168860] Key type .llcrypt registered [ 939.470240] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:10:12 (1783951812) [ 939.911068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 939.923222] alg: No test for adler32 (adler32-zlib) [ 940.839716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.906417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 940.922386] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 941.651179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.53 [ 941.678889] LNet: 53751:0:(config.c:1739:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 941.690901] LNet: Removed LNI 192.168.201.53@tcp [ 948.024686] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:10:21 (1783951821) [ 948.325640] LNet: 54322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.346307] LNetError: 54322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.367440] LNet: Removed LNI 192.168.201.53@tcp [ 949.044572] Key type .llcrypt unregistered [ 949.045796] Key type ._llcrypt unregistered [ 955.961495] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 956.247398] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 959.143730] Key type ._llcrypt registered [ 959.149443] Key type .llcrypt registered [ 959.304824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 959.329584] alg: No test for adler32 (adler32-zlib) [ 960.656730] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 966.747955] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:10:40 (1783951840) [ 966.944792] LNet: 55235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.951466] LNetError: 55235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 966.981958] LNet: Removed LNI 10.1.2.3@tcp [ 967.563480] Key type .llcrypt unregistered [ 967.566177] Key type ._llcrypt unregistered [ 972.314949] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 972.533058] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 972.534830] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 972.537589] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 975.295895] Key type ._llcrypt registered [ 975.303980] Key type .llcrypt registered [ 975.456969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 975.467186] alg: No test for adler32 (adler32-zlib) [ 976.439048] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 983.202418] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:10:56 (1783951856) [ 984.461908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 986.539331] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:10:59 (1783951859) [ 986.663030] LNet: 56414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.670894] LNetError: 56414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 986.691249] LNet: Removed LNI 10.1.2.3@tcp [ 987.345192] Key type .llcrypt unregistered [ 987.346198] Key type ._llcrypt unregistered [ 993.616868] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 993.821879] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 993.826864] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 993.836550] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 995.712500] Key type ._llcrypt registered [ 995.715988] Key type .llcrypt registered [ 996.028965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 996.063258] alg: No test for adler32 (adler32-zlib) [ 997.059868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.152161] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1005.162518] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:11:18 (1783951878) [ 1005.659766] LNet: 57645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.666667] LNetError: 57645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1005.685279] LNet: Removed LNI 10.1.2.3@tcp [ 1006.415559] Key type .llcrypt unregistered [ 1006.417553] Key type ._llcrypt unregistered [ 1011.992938] Key type ._llcrypt registered [ 1012.000739] Key type .llcrypt registered [ 1012.264807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1012.303969] alg: No test for adler32 (adler32-zlib) [ 1013.326475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1013.370648] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1022.850906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.153@tcp [ 1023.630199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_interrupt -r 1 [ 1023.671664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_interrupt -r 1 [ 1023.713734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1023.727765] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1023.938937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1023.966961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1023.997334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1024.053473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1024.109484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1024.158609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1024.213286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1024.271870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1024.305337] LNet: 58083:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1024.318512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1024.350256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1024.386357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1024.460237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1024.522645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1024.588740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1024.667576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1025.434670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_dropped -r 1 [ 1025.500427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_dropped -r 1 [ 1025.579445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1025.598383] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1025.698621] LNet: 1 local NIs in recovery (showing 1): 192.168.201.53@tcp [ 1025.935430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1025.986610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1026.018137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1026.056253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1026.086256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1026.132967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1026.185361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1026.218656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1026.251856] LNet: 58192:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1026.270474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1026.315756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1026.346917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1026.381782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1026.419242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1026.477600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1026.528287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1027.289625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_aborted -r 1 [ 1027.353291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_aborted -r 1 [ 1027.381432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1027.406667] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1027.678615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1027.707307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1027.730717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1027.760224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1027.785324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1027.815245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1027.843086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1027.877322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1027.899335] LNet: 58301:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1027.937294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1027.992911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1028.033433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1028.083371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1028.129705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1028.176844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1028.234505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1028.763310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_no_route -r 1 [ 1028.800608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_no_route -r 1 [ 1028.828129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1029.060932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1029.086970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1029.120502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1029.149755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1029.173925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1029.213618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1029.259691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1029.307364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1029.349634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1029.378939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1029.407641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1029.435148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1029.470597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1029.513839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1029.568174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1030.188522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_timeout -r 1 [ 1030.214523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_timeout -r 1 [ 1030.240091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1030.261806] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1030.268283] LNet: Skipped 1 previous similar message [ 1030.532958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1030.559267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1030.584879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1030.611966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1030.656516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1030.708165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1030.743115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1030.804853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1030.828961] LNet: 58520:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1030.834817] LNet: 58520:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1030.857907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1030.904784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1030.947794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1030.972573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1031.008418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1031.050788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1031.086777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1031.637250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_error -r 1 [ 1031.677642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_error -r 1 [ 1031.723847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1031.987765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1032.034859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1032.072723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1032.101613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1032.152840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1032.197504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1032.232886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1032.268576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1032.311893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1032.363227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1032.411159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1032.466437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1032.520899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1032.555537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1032.615326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1032.875896] LNet: 58670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.887411] LNetError: 58670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1033.956778] LNet: Removed LNI 192.168.201.53@tcp [ 1034.462899] Key type .llcrypt unregistered [ 1034.471546] Key type ._llcrypt unregistered [ 1052.408316] Key type ._llcrypt registered [ 1052.410822] Key type .llcrypt registered [ 1053.878361] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:12:06 (1783951926) [ 1054.679432] Key type .llcrypt unregistered [ 1054.681883] Key type ._llcrypt unregistered [ 1061.590786] Key type ._llcrypt registered [ 1061.603335] Key type .llcrypt registered [ 1061.864977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1061.884508] alg: No test for adler32 (adler32-zlib) [ 1062.820945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1062.853961] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1070.200582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.153@tcp [ 1081.313590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.389228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1081.446773] LNet: Added LNI 192.168.201.53@tcp1 [8/256/0/180] [ 1082.872498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_interrupt -r 1 [ 1082.920606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp1 -m GET -e local_interrupt -r 1 [ 1082.962174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_interrupt -r 1 [ 1083.009663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -m GET -e local_interrupt -r 1 [ 1083.075538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp -m GET -e local_interrupt -r 1 [ 1083.110806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp1 -m GET -e local_interrupt -r 1 [ 1083.141374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -m GET -e local_interrupt -r 1 [ 1083.178892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -m GET -e local_interrupt -r 1 [ 1083.205768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1083.222367] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1083.228715] LNet: 1 local NIs in recovery (showing 1): 192.168.201.53@tcp [ 1083.499768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1083.525714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1083.544638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1083.571724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1083.602402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1083.634627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1083.657764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1083.678478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1083.687968] LNet: 60140:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1083.701796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1083.729245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1083.753460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1083.780150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1083.806792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1083.830756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1083.853968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1084.252343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_dropped -r 1 [ 1084.301769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp1 -m GET -e local_dropped -r 1 [ 1084.333098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_dropped -r 1 [ 1084.395415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -m GET -e local_dropped -r 1 [ 1084.442420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp -m GET -e local_dropped -r 1 [ 1084.472976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp1 -m GET -e local_dropped -r 1 [ 1084.500977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -m GET -e local_dropped -r 1 [ 1084.534714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -m GET -e local_dropped -r 1 [ 1084.576920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1084.594093] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1084.602714] LNet: Skipped 2 previous similar messages [ 1084.793858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1084.819264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1084.843275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1084.864273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1084.888560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1084.904916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1084.924752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1084.947986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1084.971789] LNet: 60262:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1084.989755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1085.021579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1085.047848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1085.071983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1085.101607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1085.127955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1085.155316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1085.623761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_aborted -r 1 [ 1085.654188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp1 -m GET -e local_aborted -r 1 [ 1085.702301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_aborted -r 1 [ 1085.746317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -m GET -e local_aborted -r 1 [ 1085.781191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp -m GET -e local_aborted -r 1 [ 1085.813247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp1 -m GET -e local_aborted -r 1 [ 1085.845356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -m GET -e local_aborted -r 1 [ 1085.887983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -m GET -e local_aborted -r 1 [ 1085.921667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1085.936536] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1085.941637] LNet: Skipped 2 previous similar messages [ 1086.137861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1086.165645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1086.194871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1086.227831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1086.246583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1086.274288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1086.343641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1086.391372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1086.428023] LNet: 60384:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1086.454381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1086.496412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1086.556883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1086.606067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1086.656414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1086.705182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1086.735371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1087.354586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_no_route -r 1 [ 1087.391855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp1 -m GET -e local_no_route -r 1 [ 1087.447726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_no_route -r 1 [ 1087.500617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -m GET -e local_no_route -r 1 [ 1087.562848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp -m GET -e local_no_route -r 1 [ 1087.603642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp1 -m GET -e local_no_route -r 1 [ 1087.649537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -m GET -e local_no_route -r 1 [ 1087.693958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -m GET -e local_no_route -r 1 [ 1087.740072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1088.024762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1088.075774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1088.113951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1088.163264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1088.212473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1088.245472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1088.295391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1088.328486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1088.359860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1088.391783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1088.432404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1088.478737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1088.515485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1088.571594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1088.635731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1089.275680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_timeout -r 1 [ 1089.305943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp1 -m GET -e local_timeout -r 1 [ 1089.344958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_timeout -r 1 [ 1089.376300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -m GET -e local_timeout -r 1 [ 1089.407444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp -m GET -e local_timeout -r 1 [ 1089.447462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp1 -m GET -e local_timeout -r 1 [ 1089.474962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -m GET -e local_timeout -r 1 [ 1089.507291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -m GET -e local_timeout -r 1 [ 1089.540695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1089.561615] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1089.564906] LNet: Skipped 5 previous similar messages [ 1089.772626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1089.794388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1089.834950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1089.877913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1089.916387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1089.946555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1089.990970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1090.018502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1090.035261] LNet: 60626:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1090.041608] LNet: 60626:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1090.059451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1090.087765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1090.120671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1090.160635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1090.206758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1090.239398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1090.278954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1090.754607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_error -r 1 [ 1090.782950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp1 -m GET -e local_error -r 1 [ 1090.809349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_error -r 1 [ 1090.839793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -m GET -e local_error -r 1 [ 1090.867830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp -m GET -e local_error -r 1 [ 1090.894862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp1 -m GET -e local_error -r 1 [ 1090.926654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -m GET -e local_error -r 1 [ 1090.955791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -m GET -e local_error -r 1 [ 1090.981481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1091.157714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1091.187087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1091.210820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1091.243568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1091.287838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1091.334961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1091.364627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1091.397513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1091.428378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1091.454562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1091.484186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1091.504525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1091.518804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1091.537700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1091.573461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1096.108337] LNet: 60824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1096.113445] LNetError: 60824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1096.121244] LNet: Removed LNI 192.168.201.53@tcp [ 1096.130631] LNet: Removed LNI 192.168.201.53@tcp1 [ 1096.511549] Key type .llcrypt unregistered [ 1096.514436] Key type ._llcrypt unregistered [ 1110.936509] Key type ._llcrypt registered [ 1110.939537] Key type .llcrypt registered [ 1112.359225] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:13:05 (1783951985) [ 1113.098648] Key type .llcrypt unregistered [ 1113.102048] Key type ._llcrypt unregistered [ 1119.647139] Key type ._llcrypt registered [ 1119.651087] Key type .llcrypt registered [ 1120.055508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1120.067663] alg: No test for adler32 (adler32-zlib) [ 1121.099645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1121.136293] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1128.272814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.153@tcp [ 1129.003356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e remote_dropped -r 1 [ 1129.067938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e remote_dropped -r 1 [ 1129.103959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1129.123857] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1129.408286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1129.438429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1129.466928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1129.496726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1129.529327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1129.562279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1129.588351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1129.616446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1129.631894] LNet: 62072:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1129.648418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1129.685993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1129.712220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1129.748779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1129.794321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1129.844536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1129.890903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1130.559729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e remote_error -r 1 [ 1130.604128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e remote_error -r 1 [ 1130.659703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1130.694917] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1130.978232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1131.003706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1131.053515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1131.111770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1131.170130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1131.208820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1131.249141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1131.282544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1131.295064] LNet: 62181:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1131.312478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1131.335800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1131.366622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1131.398581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1131.444894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1131.507683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1131.564860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1132.301341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e remote_timeout -r 1 [ 1132.376269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e remote_timeout -r 1 [ 1132.417147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1132.442477] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1132.789448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1132.828092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1132.875853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1132.937529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1133.014970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1133.053887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1133.099993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1133.132568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.152527] LNet: 62291:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1133.184268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1133.222934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1133.254120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1133.291846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1133.325544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.397161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1133.465747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1133.760526] LNet: 62332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.773163] LNetError: 62332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.798266] LNet: Removed LNI 192.168.201.53@tcp [ 1134.409556] Key type .llcrypt unregistered [ 1134.413832] Key type ._llcrypt unregistered [ 1151.910379] Key type ._llcrypt registered [ 1151.915630] Key type .llcrypt registered [ 1153.451962] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:13:46 (1783952026) [ 1154.465463] Key type .llcrypt unregistered [ 1154.467872] Key type ._llcrypt unregistered [ 1161.240870] Key type ._llcrypt registered [ 1161.247872] Key type .llcrypt registered [ 1161.557261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1161.596973] alg: No test for adler32 (adler32-zlib) [ 1162.589895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1162.632903] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1170.563025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.153@tcp [ 1179.609244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1179.661339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1179.701814] LNet: Added LNI 192.168.201.53@tcp1 [8/256/0/180] [ 1181.426745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e remote_dropped -r 1 [ 1181.449803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp1 -m GET -e remote_dropped -r 1 [ 1181.487967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e remote_dropped -r 1 [ 1181.517930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -m GET -e remote_dropped -r 1 [ 1181.568323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp -m GET -e remote_dropped -r 1 [ 1181.604264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp1 -m GET -e remote_dropped -r 1 [ 1181.626393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -m GET -e remote_dropped -r 1 [ 1181.667530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -m GET -e remote_dropped -r 1 [ 1181.697957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1181.718584] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1181.727216] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.153@tcp [ 1181.950559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.980525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.015174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.044888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.072180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.103239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.134392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.169725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.184589] LNet: 63799:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1182.202905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.230495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.262352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.294646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.332564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.414914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.483947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1182.524626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.045981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e remote_error -r 1 [ 1183.087787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp1 -m GET -e remote_error -r 1 [ 1183.130661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e remote_error -r 1 [ 1183.171527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -m GET -e remote_error -r 1 [ 1183.211744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp -m GET -e remote_error -r 1 [ 1183.255819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp1 -m GET -e remote_error -r 1 [ 1183.303212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -m GET -e remote_error -r 1 [ 1183.388168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -m GET -e remote_error -r 1 [ 1183.427666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1183.458679] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1183.463596] LNet: Skipped 2 previous similar messages [ 1183.761690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.807521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1183.863069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.890230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.914646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.942582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.967674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.992557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.004627] LNet: 63923:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1184.022544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.060449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1184.091511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1184.128165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1184.150522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.176377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1184.204510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1184.234622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.779831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e remote_timeout -r 1 [ 1184.830450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp1 -m GET -e remote_timeout -r 1 [ 1184.882595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e remote_timeout -r 1 [ 1184.923230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -m GET -e remote_timeout -r 1 [ 1184.955960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp -m GET -e remote_timeout -r 1 [ 1185.005067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp1 -m GET -e remote_timeout -r 1 [ 1185.108021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -m GET -e remote_timeout -r 1 [ 1185.180397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -m GET -e remote_timeout -r 1 [ 1185.278679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.153@tcp [ 1185.318426] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1185.652173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1185.703664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1185.764929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1185.805721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1185.840987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1185.865400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1185.890228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1185.926539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1185.953960] LNet: 64046:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1185.982552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.015590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.044483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1186.074774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1186.105565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.139815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1186.176331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1186.199966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1191.344345] LNet: 64124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1191.357469] LNetError: 64124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.415214] LNet: Removed LNI 192.168.201.53@tcp [ 1192.424131] LNetError: 64124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.450290] LNet: Removed LNI 192.168.201.53@tcp1 [ 1193.078547] Key type .llcrypt unregistered [ 1193.081826] Key type ._llcrypt unregistered [ 1210.810532] Key type ._llcrypt registered [ 1210.812210] Key type .llcrypt registered [ 1212.409700] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:14:45 (1783952085) [ 1214.149473] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1216.010588] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:14:49 (1783952089) [ 1216.968421] Key type .llcrypt unregistered [ 1216.970074] Key type ._llcrypt unregistered [ 1223.527895] Key type ._llcrypt registered [ 1223.530579] Key type .llcrypt registered [ 1223.809957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1223.834491] alg: No test for adler32 (adler32-zlib) [ 1224.776631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1224.805166] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1232.866870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.153@tcp [ 1233.483510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e network_timeout -r 1 [ 1233.520992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e network_timeout -r 1 [ 1233.551222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.153@tcp [ 1233.584441] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1233.807851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1233.836911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1233.874725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1233.905461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1233.938916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1233.962542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1233.985925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1234.015284] LNet: 1 local NIs in recovery (showing 1): 192.168.201.53@tcp [ 1234.025913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1234.048771] LNet: 65670:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1234.063493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1234.090828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.143310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1234.189265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1234.219992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1234.257286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1234.588265] LNet: 65709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1234.595779] LNetError: 65709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1235.622734] LNet: Removed LNI 192.168.201.53@tcp [ 1236.068689] Key type .llcrypt unregistered [ 1236.070264] Key type ._llcrypt unregistered [ 1257.936355] Key type ._llcrypt registered [ 1257.940841] Key type .llcrypt registered [ 1258.153377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1258.163280] alg: No test for adler32 (adler32-zlib) [ 1259.100095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1259.139964] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1266.899124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.153@tcp [ 1275.973935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1276.035385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1276.072431] LNet: Added LNI 192.168.201.53@tcp1 [8/256/0/180] [ 1277.445734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e network_timeout -r 1 [ 1277.499263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp1 -m GET -e network_timeout -r 1 [ 1277.566744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e network_timeout -r 1 [ 1277.635501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -m GET -e network_timeout -r 1 [ 1277.711490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp -m GET -e network_timeout -r 1 [ 1277.796230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp1 -m GET -e network_timeout -r 1 [ 1277.862319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -m GET -e network_timeout -r 1 [ 1277.916500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -m GET -e network_timeout -r 1 [ 1277.973167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.153@tcp [ 1278.010383] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1278.299982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1278.329988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1278.355907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1278.385699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1278.436802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1278.459623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.482410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1278.505751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.522550] LNet: 66590:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.542696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.576945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.609683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.625381] LNet: 1 local NIs in recovery (showing 1): 192.168.201.53@tcp [ 1278.636469] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.153@tcp [ 1278.668387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.689827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.718507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1284.220849] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1284.611593] LNet: 66664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1284.621402] LNetError: 66664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1284.630675] LNet: Removed LNI 192.168.201.53@tcp [ 1284.643097] LNet: Removed LNI 192.168.201.53@tcp1 [ 1285.296727] Key type .llcrypt unregistered [ 1285.302388] Key type ._llcrypt unregistered [ 1303.917412] Key type ._llcrypt registered [ 1303.920171] Key type .llcrypt registered [ 1305.510316] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:16:18 (1783952178) [ 1306.063457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1306.084683] alg: No test for adler32 (adler32-zlib) [ 1307.108961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1307.156945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1307.186542] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1307.909927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1307.927411] LNet: Added LNI 192.168.201.53@tcp1 [8/256/0/180] [ 1308.860465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 1308.931326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1308.966448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1308.999041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1309.034508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1309.080894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1309.117266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.53@tcp [ 1309.156759] LNetError: 67767:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.201.53@tcp to recovery [ 1309.218103] LNet: 1 local NIs in recovery (showing 1): 192.168.201.53@tcp [ 1309.222734] LNet: There was an unexpected network error while writing to 192.168.201.53: rc = -22 [ 1311.263600] LNet: There was an unexpected network error while writing to 192.168.201.53: rc = -22 [ 1311.511347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1315.359710] LNet: There was an unexpected network error while writing to 192.168.201.53: rc = -22 [ 1316.230470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1316.277159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1316.341511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1316.372220] LNet: 67875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1316.384027] LNetError: 67875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1317.408616] LNet: Removed LNI 192.168.201.53@tcp [ 1317.412304] LNetError: 67875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1317.456149] LNet: Removed LNI 192.168.201.53@tcp1 [ 1317.478838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1317.562191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1317.588377] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1318.347975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1318.386378] LNet: Added LNI 192.168.201.53@tcp1 [8/256/0/180] [ 1319.262900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 1319.333706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1319.424567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1319.508468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1319.565926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1319.612893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1319.658238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1319.685162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.53@tcp [ 1319.707939] LNetError: 68147:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.201.53@tcp to recovery [ 1320.607375] LNet: There was an unexpected network error while writing to 192.168.201.53: rc = -22 [ 1323.357559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1326.757562] LNet: There was an unexpected network error while writing to 192.168.201.53: rc = -22 [ 1326.768225] LNet: Skipped 1 previous similar message [ 1331.322793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1331.359578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1331.395561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1338.472991] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:16:51 (1783952211) [ 1338.598718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1338.663435] LNet: 68860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1338.679754] LNetError: 68860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1338.693392] LNet: Removed LNI 192.168.201.53@tcp [ 1338.710626] LNet: Removed LNI 192.168.201.53@tcp1 [ 1338.739233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1338.835583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1338.861747] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1339.587857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1339.623297] LNet: Added LNI 192.168.201.53@tcp1 [8/256/0/180] [ 1340.617237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 1340.689739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1340.719355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1340.758521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1340.794511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1340.844222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1340.892246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.53@tcp --health 0 [ 1341.050733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1341.855496] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.53@tcp [ 1341.857145] LNet: There was an unexpected network error while writing to 192.168.201.53: rc = -22 [ 1341.870209] LNet: Skipped 1 previous similar message [ 1355.717914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1356.003639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1356.047292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1356.107623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.53@tcp --health 500 [ 1356.244178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1363.304231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1363.390209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1363.425588] LNet: 69349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1363.441740] LNetError: 69349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1363.453326] LNetError: 69349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1363.468211] LNet: Removed LNI 192.168.201.53@tcp [ 1363.485137] LNet: Removed LNI 192.168.201.53@tcp1 [ 1363.518062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1363.608368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1363.668447] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1364.582187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1364.607699] LNet: Added LNI 192.168.201.53@tcp1 [8/256/0/180] [ 1365.916387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 1365.956557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1366.008683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1366.050639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1366.137926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1366.204482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1366.240658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1366.277841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.53@tcp --health 0 [ 1367.649421] LNet: There was an unexpected network error while writing to 192.168.201.53: rc = -22 [ 1367.660500] LNet: Skipped 3 previous similar messages [ 1368.873549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1378.374591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1378.412922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1378.466980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1386.647996] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:17:39 (1783952259) [ 1387.221900] LNet: 70397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1387.229485] LNetError: 70397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1387.241556] LNetError: 70397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1387.250608] LNet: Removed LNI 192.168.201.53@tcp [ 1387.270097] LNet: Removed LNI 192.168.201.53@tcp1 [ 1387.948068] Key type .llcrypt unregistered [ 1387.962925] Key type ._llcrypt unregistered [ 1394.160360] Key type ._llcrypt registered [ 1394.162472] Key type .llcrypt registered [ 1394.433598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1394.448627] alg: No test for adler32 (adler32-zlib) [ 1395.409520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1395.466523] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1404.946557] Lustre: DEBUG MARKER: Initial discovery [ 1404.980738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.153@tcp [ 1407.336833] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1407.510886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1407.573578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.153@tcp [ 1407.594455] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1407.610467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1407.662305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.153@tcp -r 1 -m GET -l 3 [ 1407.721211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.153@tcp -r 1 -m GET -e local_error [ 1407.775454] LNet: 1 local NIs in recovery (showing 1): 192.168.201.53@tcp [ 1407.778537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.153@tcp -r 1 -m PUT -l 6 [ 1409.369747] Lustre: DEBUG MARKER: Force oleg153-client.virtnet to discover 192.168.201.153@tcp (in background) [ 1409.489058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.153@tcp [ 1410.755941] Lustre: DEBUG MARKER: Force oleg153-server to discover 192.168.201.53@tcp [ 1412.580042] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1413.438554] Lustre: DEBUG MARKER: Wait for 71154 [ 1415.149432] Lustre: DEBUG MARKER: Finished wait on 71154 [ 1415.513091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1415.560622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1415.565962] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1415.800384] LNet: 71609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1415.812949] LNetError: 71609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1416.870820] LNet: Removed LNI 192.168.201.53@tcp [ 1417.371629] Key type .llcrypt unregistered [ 1417.376144] Key type ._llcrypt unregistered [ 1434.463965] Key type ._llcrypt registered [ 1434.466495] Key type .llcrypt registered [ 1436.059271] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:18:28 (1783952308) [ 1436.613516] Key type .llcrypt unregistered [ 1436.616690] Key type ._llcrypt unregistered [ 1443.773505] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1443.966739] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1443.982141] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1443.993726] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1446.581600] Key type ._llcrypt registered [ 1446.583680] Key type .llcrypt registered [ 1446.726487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1446.748867] alg: No test for adler32 (adler32-zlib) [ 1447.725863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1447.780666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1447.804596] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1448.705478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1448.739988] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1457.951282] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 10:18:50 (1783952330) [ 1458.082827] LNet: 73675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1458.105518] LNetError: 73675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1458.115496] LNet: Removed LNI 10.1.2.3@tcp [ 1458.132692] LNet: Removed LNI 192.168.201.53@tcp [ 1458.703808] Key type .llcrypt unregistered [ 1458.705744] Key type ._llcrypt unregistered [ 1464.742438] Key type ._llcrypt registered [ 1464.746628] Key type .llcrypt registered [ 1464.952656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1464.972176] alg: No test for adler32 (adler32-zlib) [ 1465.851773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1465.894510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1465.923358] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1466.374773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1466.393459] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1467.207558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.53@tcp [ 1468.258568] LNetError: Unexpected error -22 connecting to 192.168.201.53@tcp at host 192.168.201.53:988 [ 1473.195505] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 10:19:06 (1783952346) [ 1473.466475] LNet: 74813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1473.477335] LNetError: 74813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1473.484971] LNet: Removed LNI 10.1.2.3@tcp [ 1473.498233] LNet: Removed LNI 192.168.201.53@tcp [ 1474.322421] Key type .llcrypt unregistered [ 1474.324626] Key type ._llcrypt unregistered [ 1481.509604] Key type ._llcrypt registered [ 1481.511889] Key type .llcrypt registered [ 1481.846545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1481.865573] alg: No test for adler32 (adler32-zlib) [ 1482.824378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1482.864527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1482.883153] LNet: Added LNI 192.168.201.53@tcp1 [8/256/0/180] [ 1483.689223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1483.721839] LNet: Added LNI 192.168.201.53@tcp2 [8/256/0/180] [ 1484.606993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.53@tcp1 --nid 192.168.201.53@tcp2 [ 1488.600879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.53@tcp1 192.168.201.53@tcp2 [ 1488.637696] LNetError: 75647:0:(lib-move.c:2063:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.53@tcp1 to 192.168.201.53@tcp2 - Route table may be misconfigured [ 1488.671343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.53@tcp2 192.168.201.53@tcp1 [ 1496.444673] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 10:19:29 (1783952369) [ 1496.641349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1496.691135] LNet: 76219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1496.703268] LNetError: 76219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1496.716620] LNet: Removed LNI 192.168.201.53@tcp1 [ 1497.778015] LNet: Removed LNI 192.168.201.53@tcp2 [ 1497.832237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1497.911865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1497.947058] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1498.701640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1498.718791] LNet: Added LNI 192.168.201.53@tcp1 [8/256/0/180] [ 1500.046934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 1500.077873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.53@tcp 192.168.201.53@tcp [ 1500.122829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.53@tcp1 192.168.201.53@tcp1 [ 1500.214635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.53@tcp -d 192.168.201.53@tcp -e network_timeout [ 1500.266445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -e network_timeout [ 1500.330415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -e network_timeout [ 1500.396164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -e network_timeout [ 1500.444355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 1500.472455] LNet: There was an unexpected network error while writing to 192.168.201.53: rc = -22 [ 1500.493600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1500.596546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1500.719647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1500.978299] LNet: 1 local NIs in recovery (showing 1): 192.168.201.53@tcp [ 1507.589495] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 10:19:40 (1783952380) [ 1507.715673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1507.752380] LNet: 77076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1507.766108] LNetError: 77076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1507.768932] LNetError: 77076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1507.802153] LNet: Removed LNI 192.168.201.53@tcp [ 1508.842343] LNet: Removed LNI 192.168.201.53@tcp1 [ 1508.906937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1509.058567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1509.389702] LNet: 77121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.831594] Key type .llcrypt unregistered [ 1509.832788] Key type ._llcrypt unregistered [ 1521.473697] Key type ._llcrypt registered [ 1521.480836] Key type .llcrypt registered [ 1522.915488] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 10:19:55 (1783952395) [ 1523.285399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1523.297640] alg: No test for adler32 (adler32-zlib) [ 1524.262434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.656695] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1527.367243] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 10:20:00 (1783952400) [ 1527.695622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1527.733225] LNet: 78218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1527.753224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.785360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1527.803405] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1528.624366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1528.644598] LNet: Added LNI 192.168.201.53@tcp1 [8/256/0/180] [ 1529.609909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 1529.641237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp1 [ 1529.699646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp1 [ 1536.504501] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 10:20:09 (1783952409) [ 1538.536570] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg153-server [ 1540.139418] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 10:20:13 (1783952413) [ 1541.992438] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg153-server [ 1543.630337] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 10:20:16 (1783952416) [ 1545.656906] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg153-server [ 1547.152320] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 10:20:20 (1783952420) [ 1549.390487] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg153-server [ 1551.050760] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 10:20:24 (1783952424) [ 1552.993862] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg153-server [ 1554.733856] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 10:20:27 (1783952427) [ 1556.544624] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg153-server [ 1558.147063] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 10:20:31 (1783952431) [ 1560.408360] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg153-server [ 1562.711900] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 10:20:35 (1783952435) [ 1565.098926] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg153-server [ 1566.807594] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 10:20:39 (1783952439) [ 1569.166833] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg153-server [ 1570.974094] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 10:20:43 (1783952443) [ 1571.090335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1571.165590] LNet: 82089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.181885] LNetError: 82089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1572.191427] LNet: Removed LNI 192.168.201.53@tcp [ 1572.211855] LNetError: 82089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1572.230280] LNet: Removed LNI 192.168.201.53@tcp1 [ 1572.265801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.384271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1572.408556] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1573.266299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1573.354932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 1573.592288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.628568] LNet: 82244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1573.641800] LNetError: 82244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.671363] LNet: Removed LNI 192.168.201.53@tcp [ 1573.736764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.821549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1573.870406] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1574.647538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 1574.835574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.914486] LNet: Removed LNI 192.168.201.53@tcp [ 1575.940031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.017536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1576.041912] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1576.900886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1576.983642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 1577.149349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.169336] LNet: 82536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.176980] LNet: 82536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1577.189437] LNetError: 82536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1577.204162] LNetError: 82536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1577.221466] LNet: Removed LNI 192.168.201.53@tcp [ 1577.253015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.329582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1577.370795] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1577.879801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 1578.042674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.118713] LNet: Removed LNI 192.168.201.53@tcp [ 1579.171356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.236743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1579.272367] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1579.823233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1579.884500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 1580.111578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.157380] LNet: Removed LNI 192.168.201.53@tcp [ 1581.201119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.286874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1581.309993] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1582.309097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 1582.556772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.600588] LNet: 82974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1582.604629] LNet: 82974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1582.613042] LNetError: 82974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1582.618804] LNetError: 82974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1582.640854] LNet: Removed LNI 192.168.201.53@tcp [ 1582.678709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.802472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1582.859371] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1583.917892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1584.035831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 1584.389823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.451538] LNet: Removed LNI 192.168.201.53@tcp [ 1585.488651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.566552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1585.602777] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1586.632928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 1586.871593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.942564] LNet: Removed LNI 192.168.201.53@tcp [ 1587.994184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.094738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1588.129344] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1588.930325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1589.032378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 1589.378845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1590.439272] LNet: Removed LNI 192.168.201.53@tcp [ 1590.503520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.603212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1590.644250] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1591.550653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 1591.884545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.921085] LNet: 83559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.928479] LNet: 83559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1591.943611] LNetError: 83559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1591.950979] LNetError: 83559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1591.975705] LNet: Removed LNI 192.168.201.53@tcp [ 1592.042878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.134844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1592.195040] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1593.357823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1593.499366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 1595.017372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.074317] LNet: Removed LNI 192.168.201.53@tcp [ 1596.137769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.220719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1596.276484] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1597.371447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 1597.682921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.759480] LNet: Removed LNI 192.168.201.53@tcp [ 1598.814552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.895765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1598.934647] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1599.822133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1599.896925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1599.926031] LNet: Removed LNI 192.168.201.53@tcp [ 1599.959764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.054428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1600.091953] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1600.952596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1601.037515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.078656] LNet: Removed LNI 192.168.201.53@tcp [ 1601.109296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.153886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1601.248541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.296417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.379263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1601.477240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.510791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.578173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1601.660236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.685374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.754137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1601.777205] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1602.769908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.834963] LNet: Removed LNI 192.168.201.53@tcp [ 1602.881721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.014629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1610.759799] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 10:21:23 (1783952483) [ 1610.939863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.983032] LNet: 84876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.986724] LNet: 84876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1611.024262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.125295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1611.149789] LNet: Added LNI 192.168.201.53@tcp231 [8/256/0/180] [ 1612.394929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.419852] LNetError: 85021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.426746] LNetError: 85021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1612.444538] LNet: Removed LNI 192.168.201.53@tcp231 [ 1612.473646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.600356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1612.650196] LNet: Added LNI 192.168.201.53@tcp231 [8/256/0/180] [ 1612.666466] LNet: 85029:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1613.437155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1613.463702] LNet: Removed LNI 192.168.201.53@tcp231 [ 1613.490353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1613.538570] LNet: Added LNI 192.168.201.53@tcp231 [8/256/0/180] [ 1615.293957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.409403] LNet: Removed LNI 192.168.201.53@tcp231 [ 1615.441651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.552853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1615.581720] LNet: Added LNI 192.168.201.53@tcp231 [8/256/0/0] [ 1615.590916] LNet: 85182:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1616.185170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1616.233641] LNet: Removed LNI 192.168.201.53@tcp231 [ 1616.250472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1616.304861] LNet: Added LNI 192.168.201.53@tcp231 [8/256/0/0] [ 1618.039707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1618.066652] LNet: Removed LNI 192.168.201.53@tcp231 [ 1618.089381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.172764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1618.194173] LNet: Added LNI 192.168.201.53@tcp231 [8/256/0/60] [ 1618.199177] LNet: 85333:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1618.771247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1618.835321] LNet: Removed LNI 192.168.201.53@tcp231 [ 1618.859595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1618.922793] LNet: Added LNI 192.168.201.53@tcp231 [8/256/0/60] [ 1620.015505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.041493] LNet: Removed LNI 192.168.201.53@tcp231 [ 1620.058342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.127720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1620.162783] LNet: Added LNI 192.168.201.53@tcp231 [8/256/0/180] [ 1620.505599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1620.539072] LNet: Removed LNI 192.168.201.53@tcp231 [ 1620.566270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1620.609593] LNet: Added LNI 192.168.201.53@tcp231 [8/256/0/180] [ 1628.928471] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 10:21:42 (1783952502) [ 1629.171589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1629.203089] LNet: Removed LNI 192.168.201.53@tcp231 [ 1629.223858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.300355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1629.327559] LNet: Added LNI 192.168.201.53@tcp232 [8/256/0/180] [ 1629.957533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1629.990587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1630.016694] LNet: Removed LNI 192.168.201.53@tcp232 [ 1630.037768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1630.055902] LNet: Added LNI 192.168.201.53@tcp232 [8/256/0/180] [ 1630.666828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1630.702383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1630.745640] LNet: Removed LNI 192.168.201.53@tcp232 [ 1630.770158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1630.794527] LNet: Added LNI 192.168.201.53@tcp232 [8/256/0/180] [ 1631.605705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1631.651717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1631.695415] LNet: Removed LNI 192.168.201.53@tcp232 [ 1631.728754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1631.770873] LNet: Added LNI 192.168.201.53@tcp232 [8/256/0/180] [ 1632.612504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1632.689820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1632.714620] LNet: Removed LNI 192.168.201.53@tcp232 [ 1632.733688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1638.789510] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 10:21:51 (1783952511) [ 1639.902400] Key type .llcrypt unregistered [ 1639.911084] Key type ._llcrypt unregistered [ 1646.442056] Key type ._llcrypt registered [ 1646.443352] Key type .llcrypt registered [ 1646.653417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1646.673506] alg: No test for adler32 (adler32-zlib) [ 1647.602279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1647.641118] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1655.959141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.153@tcp [ 1665.283641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.356631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1665.378895] LNet: Added LNI 192.168.201.53@tcp1 [8/256/0/180] [ 1667.161858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_interrupt -r 2 [ 1667.203688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp1 -m GET -e local_interrupt -r 2 [ 1667.249279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_interrupt -r 2 [ 1667.294900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -m GET -e local_interrupt -r 2 [ 1667.343143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp -m GET -e local_interrupt -r 2 [ 1667.392933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp1 -m GET -e local_interrupt -r 2 [ 1667.448233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -m GET -e local_interrupt -r 2 [ 1667.493160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -m GET -e local_interrupt -r 2 [ 1667.533445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.53@tcp 192.168.201.153@tcp [ 1667.577431] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1667.593552] LNet: 1 local NIs in recovery (showing 1): 192.168.201.53@tcp [ 1667.972529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1668.023564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1668.068165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1668.104650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1668.137795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1668.201723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1668.255371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1668.320461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1668.379865] LNet: 87987:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1668.414150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1668.481319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1668.534971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1668.573937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1668.638929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1668.703620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1669.303442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_dropped -r 2 [ 1669.348571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp1 -m GET -e local_dropped -r 2 [ 1669.416570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_dropped -r 2 [ 1669.475113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -m GET -e local_dropped -r 2 [ 1669.550256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp -m GET -e local_dropped -r 2 [ 1669.622846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp1 -m GET -e local_dropped -r 2 [ 1669.686678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -m GET -e local_dropped -r 2 [ 1669.745206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -m GET -e local_dropped -r 2 [ 1669.807509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.53@tcp 192.168.201.153@tcp [ 1669.838566] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1670.288337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1670.362064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1670.430433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1670.506537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1670.596815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1670.665543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1670.731723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1670.806104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1670.846589] LNet: 88108:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1670.868463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1670.900183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1670.936593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1670.964498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1670.999395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1671.061636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1671.562935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_aborted -r 2 [ 1671.607762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp1 -m GET -e local_aborted -r 2 [ 1671.638275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_aborted -r 2 [ 1671.688749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -m GET -e local_aborted -r 2 [ 1671.745964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp -m GET -e local_aborted -r 2 [ 1671.782655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp1 -m GET -e local_aborted -r 2 [ 1671.814977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -m GET -e local_aborted -r 2 [ 1671.850945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -m GET -e local_aborted -r 2 [ 1671.880680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.53@tcp 192.168.201.153@tcp [ 1671.899942] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1672.144679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1672.175869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1672.216522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1672.241392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1672.267449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1672.305323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1672.346946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1672.388711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1672.406920] LNet: 88228:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1672.428953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1672.490056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1672.528499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1672.567963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1672.609908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1672.654134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1673.121583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_no_route -r 2 [ 1673.138150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp1 -m GET -e local_no_route -r 2 [ 1673.155404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_no_route -r 2 [ 1673.173984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -m GET -e local_no_route -r 2 [ 1673.204802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp -m GET -e local_no_route -r 2 [ 1673.247812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp1 -m GET -e local_no_route -r 2 [ 1673.266981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -m GET -e local_no_route -r 2 [ 1673.287127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -m GET -e local_no_route -r 2 [ 1673.316518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.53@tcp 192.168.201.153@tcp [ 1673.498721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1673.511982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1673.529628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1673.551508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1673.575117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1673.596829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1673.613676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1673.631569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1673.649826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1673.672335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1673.696911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1673.720238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1673.766906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1673.794259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1674.308604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e local_timeout -r 2 [ 1674.355357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp1 -m GET -e local_timeout -r 2 [ 1674.388479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e local_timeout -r 2 [ 1674.429868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -m GET -e local_timeout -r 2 [ 1674.459150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp -m GET -e local_timeout -r 2 [ 1674.495507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp1 -m GET -e local_timeout -r 2 [ 1674.531802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -m GET -e local_timeout -r 2 [ 1674.555800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -m GET -e local_timeout -r 2 [ 1674.581469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.53@tcp 192.168.201.153@tcp [ 1674.598416] LNet: There was an unexpected network error while writing to 192.168.201.153: rc = -22 [ 1674.604718] LNet: Skipped 1 previous similar message [ 1674.758846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1674.782916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1674.804746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1674.824828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1674.845457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1674.865738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1674.888408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1674.912373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1674.923942] LNet: 88469:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1674.926601] LNet: 88469:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1674.937967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1674.956522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1674.982134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1675.008980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1675.037798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1675.063464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1675.680965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp -m GET -e remote_dropped -r 2 [ 1675.753610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.153@tcp1 -m GET -e remote_dropped -r 2 [ 1675.819513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp -m GET -e remote_dropped -r 2 [ 1675.878331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp -d 192.168.201.53@tcp1 -m GET -e remote_dropped -r 2 [ 1675.933325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp -m GET -e remote_dropped -r 2 [ 1675.999542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.153@tcp1 -m GET -e remote_dropped -r 2 [ 1676.043717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp -m GET -e remote_dropped -r 2 [ 1676.080546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.53@tcp1 -d 192.168.201.53@tcp1 -m GET -e remote_dropped -r 2 [ 1676.124611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.53@tcp 192.168.201.153@tcp [ 1676.144954] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.153@tcp [ 1676.401762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1676.438178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1676.483343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1676.515640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1676.560953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1676.609814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1676.666739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1676.722418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1676.793432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1676.846447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1676.899903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1676.936876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1676.964566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1676.990508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1682.508223] LNet: 88665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1682.517623] LNetError: 88665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.558574] LNet: Removed LNI 192.168.201.53@tcp [ 1683.560332] LNetError: 88665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.589809] LNet: Removed LNI 192.168.201.53@tcp1 [ 1684.368134] Key type .llcrypt unregistered [ 1684.378837] Key type ._llcrypt unregistered [ 1701.252780] Key type ._llcrypt registered [ 1701.255905] Key type .llcrypt registered [ 1702.558198] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 10:22:55 (1783952575) [ 1703.038983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1703.053723] alg: No test for adler32 (adler32-zlib) [ 1704.072630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.170789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1704.216243] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1705.312328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1705.348726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.53@tcp [ 1705.376966] LNetError: 89647:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.201.53@tcp to recovery [ 1705.478262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.53@tcp [ 1705.615835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1712.973586] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 10:23:05 (1783952585) [ 1713.254531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.289556] LNet: 90239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.301560] LNetError: 90239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.346352] LNet: Removed LNI 192.168.201.53@tcp [ 1713.388140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.475529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1713.495356] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1714.249350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1714.279854] LNet: Added LNI 192.168.201.53@tcp1 [8/256/0/180] [ 1715.230562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.53@tcp [ 1715.268385] LNetError: 90505:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.201.53@tcp to recovery [ 1715.487682] LNet: 1 local NIs in recovery (showing 1): 192.168.201.53@tcp [ 1715.489197] LNet: There was an unexpected network error while writing to 192.168.201.53: rc = -22 [ 1716.592772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.53@tcp [ 1724.464194] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 10:23:17 (1783952597) [ 1726.526732] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg153-server [ 1728.283994] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 10:23:21 (1783952601) [ 1730.273940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg153-server [ 1732.370108] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 10:23:25 (1783952605) [ 1734.413370] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg153-server [ 1736.306321] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:23:29 (1783952609) [ 1736.401748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.417586] LNet: 92121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.422737] LNetError: 92121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.429440] LNet: Removed LNI 192.168.201.53@tcp [ 1736.438325] LNet: Removed LNI 192.168.201.53@tcp1 [ 1736.462514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.585239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1736.612563] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1737.332900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1737.412412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1737.480886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1737.585199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1737.634620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1737.670634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1744.796375] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 10:23:37 (1783952617) [ 1744.984547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.026948] LNet: 92852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.049413] LNetError: 92852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.062672] LNetError: 92852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1745.101364] LNet: Removed LNI 192.168.201.53@tcp [ 1745.127257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.191332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1745.212973] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1752.785795] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:23:45 (1783952625) [ 1753.196632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.236625] LNet: 93558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.252618] LNetError: 93558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1753.271919] LNet: Removed LNI 192.168.201.53@tcp [ 1753.294748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.353052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1753.377700] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1761.037337] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:23:54 (1783952634) [ 1762.594107] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1764.751557] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:23:57 (1783952637) [ 1765.238417] LNet: 94592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.247431] LNetError: 94592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1765.264962] LNet: Removed LNI 192.168.201.53@tcp [ 1765.905582] Key type .llcrypt unregistered [ 1765.909587] Key type ._llcrypt unregistered [ 1772.738542] Key type ._llcrypt registered [ 1772.741888] Key type .llcrypt registered [ 1773.091145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1773.121838] alg: No test for adler32 (adler32-zlib) [ 1774.131707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1774.158400] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1782.235113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.153@tcp [ 1788.178611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.153@tcp [ 1788.696802] LNet: 95010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1788.722051] LNetError: 95010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1788.745968] LNet: Removed LNI 192.168.201.53@tcp [ 1789.286637] Key type .llcrypt unregistered [ 1789.288406] Key type ._llcrypt unregistered [ 1801.615913] Key type ._llcrypt registered [ 1801.617666] Key type .llcrypt registered [ 1803.011669] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1804.693576] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1806.494141] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:24:39 (1783952679) [ 1808.210352] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1809.792582] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:24:42 (1783952682) [ 1811.904766] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg153-server [ 1813.723852] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 10:24:46 (1783952686) [ 1815.934961] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg153-server [ 1818.165236] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:24:50 (1783952690) [ 1820.220591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1820.260780] alg: No test for adler32 (adler32-zlib) [ 1821.529861] Lustre: Lustre: Build Version: 2.17.54_162_gb5ded84 [ 1821.886713] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1823.663673] Key type lgssc registered [ 1825.503861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1844.350720] Key type lgssc unregistered [ 1844.561805] LNet: 98185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.566813] LNetError: 98185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.577869] LNet: Removed LNI 192.168.201.53@tcp [ 1845.506924] Key type .llcrypt unregistered [ 1845.510077] Key type ._llcrypt unregistered [ 1860.627313] Key type ._llcrypt registered [ 1860.629977] Key type .llcrypt registered [ 1862.264844] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 10:25:34 (1783952734) [ 1862.715989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1862.736787] alg: No test for adler32 (adler32-zlib) [ 1863.756361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1863.822900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1863.873596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1863.923772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1863.974722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1872.067531] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 10:25:44 (1783952744) [ 1872.209488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.234423] LNet: 99595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.299706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.408328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1872.450882] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1873.726538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1873.751651] LNet: 99745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.768945] LNetError: 99745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1873.785833] LNet: Removed LNI 192.168.201.53@tcp [ 1873.829988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1873.914407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1873.935109] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1874.541979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1874.569115] LNetError: 99774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.582750] LNet: Removed LNI 192.168.201.53@tcp [ 1874.602754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1874.645788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1874.665711] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1881.869249] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 10:25:54 (1783952754) [ 1882.160674] LNet: 100364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.169187] LNet: 100364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1882.177690] LNetError: 100364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.202988] LNet: Removed LNI 192.168.201.53@tcp [ 1883.079598] Key type .llcrypt unregistered [ 1883.082131] Key type ._llcrypt unregistered [ 1889.786604] Key type ._llcrypt registered [ 1889.789588] Key type .llcrypt registered [ 1890.120125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1890.159384] alg: No test for adler32 (adler32-zlib) [ 1891.201435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1891.286295] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1891.289555] LNet: *** cfs_fail_loc=e003, val=100*** [ 1892.244965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 1892.262996] LNet: *** cfs_fail_loc=e003, val=100*** [ 1892.265757] LNet: Skipped 2 previous similar messages [ 1892.272301] Ping reply received with mapping: [ 1892.275274] found metadata reply NI[192.168.201.53@tcp] [ 1892.278259] 35 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1892.283755] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1892.285951] 5a 5a 5a 5a 5a 5a 5a 5a [ 1892.515458] LNet: 100761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1892.529670] LNetError: 100761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1892.547709] LNet: Removed LNI 192.168.201.53@tcp [ 1893.405650] Key type .llcrypt unregistered [ 1893.412890] Key type ._llcrypt unregistered [ 1904.870301] Key type ._llcrypt registered [ 1904.872687] Key type .llcrypt registered [ 1906.177144] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:26:19 (1783952779) [ 1906.715386] Key type .llcrypt unregistered [ 1906.716947] Key type ._llcrypt unregistered [ 1914.559424] Key type ._llcrypt registered [ 1914.565992] Key type .llcrypt registered [ 1914.829966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1914.846708] alg: No test for adler32 (adler32-zlib) [ 1915.955259] LNetError: 101807:0:(api-ni.c:2795:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1916.299800] Key type .llcrypt unregistered [ 1916.305736] Key type ._llcrypt unregistered [ 1921.284886] Key type ._llcrypt registered [ 1921.289606] Key type .llcrypt registered [ 1922.627585] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 10:26:35 (1783952795) [ 1923.253258] Key type .llcrypt unregistered [ 1923.254936] Key type ._llcrypt unregistered [ 1929.094389] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1929.106168] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1931.826706] Key type ._llcrypt registered [ 1931.828846] Key type .llcrypt registered [ 1932.128071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1932.152027] alg: No test for adler32 (adler32-zlib) [ 1933.211784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1933.335402] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1943.354488] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 10:26:56 (1783952816) [ 1943.512717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.542399] LNet: 103581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.555569] LNetError: 103581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1943.578456] LNet: Removed LNI 10.1.2.3@tcp [ 1943.638377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1944.585226] Key type .llcrypt unregistered [ 1944.587512] Key type ._llcrypt unregistered [ 1950.880822] Key type ._llcrypt registered [ 1950.883434] Key type .llcrypt registered [ 1951.107886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1951.131370] alg: No test for adler32 (adler32-zlib) [ 1952.199097] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1952.299469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 1952.544781] LNet: 103877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1952.554859] LNetError: 103877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1953.573811] LNet: Removed LNI 192.168.201.53@tcp [ 1954.373621] Key type .llcrypt unregistered [ 1954.378850] Key type ._llcrypt unregistered [ 1966.177299] Key type ._llcrypt registered [ 1966.179297] Key type .llcrypt registered [ 1967.191467] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 10:27:20 (1783952840) [ 1968.068271] Key type .llcrypt unregistered [ 1968.070221] Key type ._llcrypt unregistered [ 1976.288608] Key type ._llcrypt registered [ 1976.290121] Key type .llcrypt registered [ 1976.629923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1976.665794] alg: No test for adler32 (adler32-zlib) [ 1985.379992] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 10:27:38 (1783952858) [ 1985.489060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1985.563305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1985.655902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1985.701547] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 1986.968778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1987.028864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1987.092299] LNet: Removed LNI 192.168.201.53@tcp [ 1987.510487] LNet: 105622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1988.050966] Key type .llcrypt unregistered [ 1988.061341] Key type ._llcrypt unregistered [ 1998.749756] Key type ._llcrypt registered [ 1998.752856] Key type .llcrypt registered [ 2000.173848] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 10:27:53 (1783952873) [ 2000.760603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2000.787144] alg: No test for adler32 (adler32-zlib) [ 2001.730436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2001.802252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2001.836546] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2002.744934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 2002.781677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.53@tcp [ 2002.801353] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2002.805192] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2002.808677] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2002, 0, 0 [ 2002.815878] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2002.821131] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2002.824915] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2002.828360] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2002, 0, 0 [ 2002.832620] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2002.835406] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2002.842435] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2002.845353] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2002, 0, 2052 [ 2002.854072] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2002.858483] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2002.861967] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2002.865435] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2002, 0, 2052 [ 2002.873453] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2009.208833] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 10:28:02 (1783952882) [ 2009.793702] LNet: 107161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.802938] LNetError: 107161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.819055] LNet: Removed LNI 192.168.201.53@tcp [ 2010.375581] Key type .llcrypt unregistered [ 2010.377683] Key type ._llcrypt unregistered [ 2017.199984] Key type ._llcrypt registered [ 2017.201846] Key type .llcrypt registered [ 2017.407915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2017.439714] alg: No test for adler32 (adler32-zlib) [ 2018.352843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2018.379862] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2025.616236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.153@tcp [ 2034.421936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2034.484199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2034.513465] LNet: Added LNI 192.168.201.53@tcp1 [8/256/0/180] [ 2035.451505] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2035.578104] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2035.583988] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2035.601080] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2037.971135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2037.987724] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2048.769292] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 10:28:42 (1783952922) [ 2048.825950] LNet: 108661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2048.843061] LNetError: 108661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2048.862396] LNet: Removed LNI 192.168.201.53@tcp [ 2049.887547] LNet: Removed LNI 192.168.201.53@tcp1 [ 2049.893817] LNetError: 108661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2049.905432] LNetError: 108661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2049.918290] LNet: Removed LNI 10.1.2.3@tcp99 [ 2050.534625] Key type .llcrypt unregistered [ 2050.537372] Key type ._llcrypt unregistered [ 2057.404641] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2057.408232] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2060.141964] Key type ._llcrypt registered [ 2060.146767] Key type .llcrypt registered [ 2060.359687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2060.373802] alg: No test for adler32 (adler32-zlib) [ 2061.311663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.387312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2061.408388] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2062.158382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2062.191427] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2064.734156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2064.819688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2064.907530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2064.962408] LNetError: 109415:0:(peer.c:2276:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2065.047159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2065.168715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2065.989936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2066.127830] LNet: 109435:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2066.133525] LNetError: 109435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2066.145597] LNet: Removed LNI 10.1.2.3@tcp [ 2067.176532] LNet: Removed LNI 192.168.201.53@tcp [ 2067.968518] Key type .llcrypt unregistered [ 2067.972549] Key type ._llcrypt unregistered [ 2081.412869] Key type ._llcrypt registered [ 2081.419748] Key type .llcrypt registered [ 2083.133689] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 10:29:15 (1783952955) [ 2083.780661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2083.809893] alg: No test for adler32 (adler32-zlib) [ 2084.822157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.905740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2084.934389] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2086.142217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.53@tcp [ 2086.211721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg153-client@tcp [ 2093.422647] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 10:29:26 (1783952966) [ 2093.497156] LNet: 110962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2093.506843] LNetError: 110962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2093.535083] LNet: Removed LNI 192.168.201.53@tcp [ 2094.316414] Key type .llcrypt unregistered [ 2094.317939] Key type ._llcrypt unregistered [ 2101.534779] Key type ._llcrypt registered [ 2101.539344] Key type .llcrypt registered [ 2101.705554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2101.726544] alg: No test for adler32 (adler32-zlib) [ 2102.819029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2102.891511] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2102.951596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2110.367327] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 10:29:43 (1783952983) [ 2111.832154] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2113.753536] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 10:29:46 (1783952986) [ 2115.508965] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2117.650158] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 10:29:50 (1783952990) [ 2118.059578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2118.109281] LNet: 112391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2118.124811] LNetError: 112391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2118.147743] LNet: Removed LNI 192.168.201.53@tcp [ 2118.177755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2118.279813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2118.348104] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2119.260056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2119.343479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2127.384690] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 10:30:00 (1783953000) [ 2127.677810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.701875] LNet: 113100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2127.711955] LNetError: 113100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2127.726225] LNet: Removed LNI 192.168.201.53@tcp [ 2127.752380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.804416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2127.832756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2128.488372] Key type .llcrypt unregistered [ 2128.490728] Key type ._llcrypt unregistered [ 2139.190355] Key type ._llcrypt registered [ 2139.191675] Key type .llcrypt registered [ 2140.589796] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 10:30:13 (1783953013) [ 2141.104744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2141.120922] alg: No test for adler32 (adler32-zlib) [ 2142.036650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2142.084598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2142.105995] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2142.650358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2142.699594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2149.070839] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 10:30:22 (1783953022) [ 2149.283456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2149.358632] LNet: 114657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2149.373492] LNetError: 114657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2150.444901] LNet: Removed LNI 192.168.201.53@tcp [ 2150.533370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.627306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2150.686992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2157.739548] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 10:30:30 (1783953030) [ 2157.909449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2157.951507] LNet: 115233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2157.995898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.090925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2158.386443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2158.435335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2158.557973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2158.957779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2158.993543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2159.053494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2159.070978] LNet: 115271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2159.092581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.174026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2159.493519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2159.521864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2159.564336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2159.984181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2160.032195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2165.517825] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 10:30:38 (1783953038) [ 2165.830617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2165.870430] LNet: 115871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.908430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2166.002373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2166.054473] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2166.957896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2167.003312] LNet: Added LNI 192.168.201.53@tcp2 [8/256/0/180] [ 2167.967267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2168.021151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2168.339565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2168.398272] LNetError: 116151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.423696] LNet: Removed LNI 192.168.201.53@tcp [ 2168.446844] LNet: Removed LNI 192.168.201.53@tcp2 [ 2168.502614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2168.604484] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2168.610575] LNet: Added LNI 192.168.201.53@tcp2 [8/256/0/180] [ 2168.619561] LNet: 116157:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2169.673960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2169.945596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2169.968554] LNet: 116194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2169.973855] LNet: 116194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2169.985695] LNetError: 116194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2169.993951] LNetError: 116194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2171.039261] LNet: Removed LNI 192.168.201.53@tcp [ 2171.052043] LNet: Removed LNI 192.168.201.53@tcp2 [ 2171.117897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2171.210454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2171.255364] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2172.256268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2172.274370] LNet: Added LNI 192.168.201.53@tcp2 [8/256/0/180] [ 2173.359693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2173.379380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2173.433687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2173.691723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2173.713805] LNetError: 116479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2173.721515] LNetError: 116479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2173.727183] LNet: Removed LNI 192.168.201.53@tcp [ 2174.758720] LNet: Removed LNI 192.168.201.53@tcp2 [ 2174.848958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2175.014170] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2175.048240] LNet: Added LNI 192.168.201.53@tcp2 [8/256/0/180] [ 2175.059906] LNet: 116485:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2175.839687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2176.096703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2176.112296] LNet: Removed LNI 192.168.201.53@tcp [ 2177.190846] LNet: Removed LNI 192.168.201.53@tcp2 [ 2177.268041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2177.461822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2177.530353] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2178.236878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2178.255504] LNet: Added LNI 192.168.201.53@tcp2 [8/256/0/180] [ 2179.041394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2179.094662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2179.393224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2179.446336] LNet: 116802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2179.456824] LNet: 116802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2179.477052] LNetError: 116802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2179.486362] LNetError: 116802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2179.500770] LNet: Removed LNI 192.168.201.53@tcp [ 2179.515141] LNet: Removed LNI 192.168.201.53@tcp2 [ 2179.542629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2179.635355] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2179.640431] LNet: Added LNI 192.168.201.53@tcp2 [8/256/0/180] [ 2179.650785] LNet: 116808:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2180.204438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2180.543928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2181.600252] LNet: Removed LNI 192.168.201.53@tcp [ 2181.619710] LNet: Removed LNI 192.168.201.53@tcp2 [ 2181.669773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.725866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2181.753852] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2182.576604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2182.597878] LNet: Added LNI 192.168.201.53@tcp2 [8/256/0/180] [ 2183.689247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2183.719568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2183.772340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2184.052411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2185.119304] LNet: Removed LNI 192.168.201.53@tcp [ 2186.149620] LNet: Removed LNI 192.168.201.53@tcp2 [ 2186.197461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.280437] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2186.294174] LNet: Added LNI 192.168.201.53@tcp2 [8/256/0/180] [ 2186.302308] LNet: 117136:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2186.926782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2187.118201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2187.134893] LNet: Removed LNI 192.168.201.53@tcp [ 2188.217055] LNet: Removed LNI 192.168.201.53@tcp2 [ 2188.265490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2188.414436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2188.456501] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2189.678715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2189.712892] LNet: Added LNI 192.168.201.53@tcp2 [8/256/0/180] [ 2190.880357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2190.935453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2191.009246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2191.332247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2191.351705] LNetError: 117470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2191.356802] LNetError: 117470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2191.367689] LNet: Removed LNI 192.168.201.53@tcp [ 2191.380605] LNet: Removed LNI 192.168.201.53@tcp2 [ 2191.409942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2191.493359] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2191.497449] LNet: Added LNI 192.168.201.53@tcp2 [8/256/0/180] [ 2191.506242] LNet: 117476:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2192.240434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2192.573210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2193.631144] LNet: Removed LNI 192.168.201.53@tcp [ 2193.645497] LNet: Removed LNI 192.168.201.53@tcp2 [ 2193.703728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.776084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2193.819236] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2194.668424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2194.693100] LNet: Added LNI 192.168.201.53@tcp2 [8/256/0/180] [ 2195.956448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2195.997289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2196.044934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2196.112652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2196.511885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2196.532343] LNet: 117830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.537135] LNet: 117830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2197.599875] LNet: Removed LNI 192.168.201.53@tcp [ 2198.638992] LNet: Removed LNI 192.168.201.53@tcp2 [ 2198.674729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2198.793092] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2198.803318] LNet: Added LNI 192.168.201.53@tcp2 [8/256/0/180] [ 2199.585868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2206.047464] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2207.327157] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 10:31:20 (1783953080) [ 2207.498838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2207.535892] LNetError: 118598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2207.540484] LNetError: 118598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2207.545591] LNet: Removed LNI 192.168.201.53@tcp [ 2208.623305] LNet: Removed LNI 192.168.201.53@tcp2 [ 2208.679470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2208.993951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2209.018194] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2209.707459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2209.730779] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2210.495968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2210.514280] LNet: Removed LNI 10.1.2.3@tcp [ 2210.527456] LNet: Removed LNI 192.168.201.53@tcp [ 2210.540991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.588448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2210.610554] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2210.614344] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2210.972137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2210.992553] LNet: Removed LNI 192.168.201.53@tcp [ 2211.006546] LNet: Removed LNI 10.1.2.3@tcp [ 2217.631282] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 10:31:30 (1783953090) [ 2217.812195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2217.887474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2218.009806] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2218.241397] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2218.247315] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2218.252852] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2220.691901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2220.721617] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2221.534555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2221.554399] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2223.366677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.53@tcp [ 2223.407740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2223.459576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.53@tcp -m PUT -e local_timeout -r 1 [ 2223.550719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2223.672164] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2223.684539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2223.918417] LNetError: 119539:0:(lib-move.c:4771:lnet_parse()) 192.168.201.53@tcp, src 192.168.201.53@tcp: Dropping PUT (error -108 looking up sender) [ 2224.935586] LNet: Removed LNI 10.1.2.3@tcp [ 2226.026531] LNet: Removed LNI 192.168.201.53@tcp [ 2226.435786] Key type .llcrypt unregistered [ 2226.439808] Key type ._llcrypt unregistered [ 2236.776077] Key type ._llcrypt registered [ 2236.777236] Key type .llcrypt registered [ 2237.996877] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 10:31:51 (1783953111) [ 2238.320869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2238.348483] alg: No test for adler32 (adler32-zlib) [ 2239.349625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.816389] LNet: 120848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2240.481386] Key type .llcrypt unregistered [ 2240.484350] Key type ._llcrypt unregistered [ 2247.806574] Key type ._llcrypt registered [ 2247.808994] Key type .llcrypt registered [ 2248.010304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2248.022484] alg: No test for adler32 (adler32-zlib) [ 2249.015056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2249.034572] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2258.292535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.153@tcp [ 2258.577836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2259.053665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.153@tcp [ 2259.245485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2259.264490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2259.280448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2259.295274] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.153@tcp [ 2259.297907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2259.321913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2259.343938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2259.365843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2259.387422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2259.402625] LNet: 121298:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2259.417529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2259.433172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2259.449507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2259.468939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2259.493569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2259.519937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2260.056842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2260.302334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2260.336728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2260.375459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2260.400347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2260.425090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2260.451938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2260.483954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2260.506898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2260.520575] LNet: 121415:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2260.535709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2260.558256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2260.588652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2260.626787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2260.665088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2260.717207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2261.479203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.153@tcp [ 2261.982197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2262.007800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2262.043538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2262.068701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2262.098791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2262.135807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2262.195220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2262.251610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2262.279681] LNet: 121532:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2262.302683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2262.357755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2262.400835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2262.441544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2262.487897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2262.541620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2263.238989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2263.553948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2263.611871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2263.633470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2263.666091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2263.702229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2263.729778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2263.756438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2263.790393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2263.817870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2263.840788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2263.873872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2263.906883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2263.944773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2263.971919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2264.248897] LNet: 121696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2264.255696] LNetError: 121696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2264.274486] LNet: Removed LNI 192.168.201.53@tcp [ 2264.737682] Key type .llcrypt unregistered [ 2264.739389] Key type ._llcrypt unregistered [ 2281.220243] Key type ._llcrypt registered [ 2281.222043] Key type .llcrypt registered [ 2282.981070] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:32:35 (1783953155) [ 2283.385292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2283.399431] alg: No test for adler32 (adler32-zlib) [ 2284.338619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.819436] LNet: 122582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2285.441667] Key type .llcrypt unregistered [ 2285.445476] Key type ._llcrypt unregistered [ 2292.391585] Key type ._llcrypt registered [ 2292.402453] Key type .llcrypt registered [ 2292.732902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2292.751982] alg: No test for adler32 (adler32-zlib) [ 2293.681397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2293.718519] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2300.045604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.153@tcp [ 2308.939412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2308.990921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2309.026456] LNet: Added LNI 192.168.201.53@tcp1 [8/256/0/180] [ 2310.180632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2310.815499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.153@tcp [ 2311.045827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2311.075447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2311.109536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2311.142164] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.153@tcp [ 2311.156882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2311.204222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2311.244187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2311.305726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2311.345329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2311.368269] LNet: 123234:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2311.391857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2311.431226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2311.470868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2311.507667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2311.547685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2311.587889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2312.289275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2312.670699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2312.726786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2312.768046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2312.793217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2312.843230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2312.892971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2312.953953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2313.004985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2313.034922] LNet: 123352:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2313.063317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2313.127166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2313.168182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2313.219267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2313.254930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2313.303610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2314.198781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.153@tcp [ 2314.611423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2314.636314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2314.670959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2314.702855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2314.731615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2314.748345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2314.778822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2314.829716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2314.873853] LNet: 123469:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2314.886522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2314.917267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2314.968257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2315.018572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2315.059889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2315.094933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2315.664737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2315.855277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2315.889265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2315.918229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2315.951148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2315.974760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2316.006676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2316.045518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2316.071584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2316.109248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2316.148318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2316.174778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2316.217547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2316.250295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2316.285034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2320.937779] LNet: 123667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2320.943089] LNetError: 123667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2320.949760] LNet: Removed LNI 192.168.201.53@tcp [ 2320.959238] LNet: Removed LNI 192.168.201.53@tcp1 [ 2321.343920] Key type .llcrypt unregistered [ 2321.347898] Key type ._llcrypt unregistered [ 2338.688793] Key type ._llcrypt registered [ 2338.691470] Key type .llcrypt registered [ 2340.100155] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 10:33:33 (1783953213) [ 2341.771950] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2343.891373] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 10:33:36 (1783953216) [ 2352.649178] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 10:33:45 (1783953225) [ 2360.219974] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 10:33:53 (1783953233) [ 2370.434269] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 10:34:02 (1783953242) [ 2380.494541] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 10:34:13 (1783953253) [ 2390.448346] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 10:34:22 (1783953262) [ 2399.955500] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 10:34:32 (1783953272) [ 2410.144419] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 10:34:42 (1783953282) [ 2419.570432] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 10:34:52 (1783953292) [ 2428.735836] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 10:35:01 (1783953301) [ 2438.638903] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 10:35:11 (1783953311) [ 2449.842551] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 10:35:22 (1783953322) [ 2459.424700] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 10:35:32 (1783953332) [ 2469.106818] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 10:35:41 (1783953341) [ 2479.025976] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 10:35:51 (1783953351) [ 2488.262914] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 10:36:01 (1783953361) [ 2497.804653] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 10:36:10 (1783953370) [ 2507.534626] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 10:36:20 (1783953380) [ 2508.023899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2508.047854] alg: No test for adler32 (adler32-zlib) [ 2509.101735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2509.376340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2509.455563] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2511.405691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2511.443844] LNet: 134673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2511.450837] LNetError: 134673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2511.488573] LNet: Removed LNI 192.168.201.53@tcp [ 2511.535433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2511.620693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2511.718237] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2521.098517] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 10:36:33 (1783953393) [ 2531.846699] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 10:36:44 (1783953404) [ 2541.502451] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 10:36:53 (1783953413) [ 2551.308230] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:37:04 (1783953424) [ 2560.500598] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:37:12 (1783953432) [ 2569.576541] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:37:22 (1783953442) [ 2578.455630] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:37:31 (1783953451) [ 2586.849695] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:37:39 (1783953459) [ 2596.356355] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:37:49 (1783953469) [ 2605.731210] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:37:58 (1783953478) [ 2614.201919] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:38:07 (1783953487) [ 2621.285127] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:38:14 (1783953494) [ 2629.763663] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2407 sec ========== 10:38:22 (1783953502) [ 2630.084725] LNet: 142142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2630.089751] LNetError: 142142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2630.103418] LNet: Removed LNI 192.168.201.53@tcp [ 2630.720705] Key type .llcrypt unregistered [ 2630.729194] Key type ._llcrypt unregistered [ 2645.172373] Key type ._llcrypt registered [ 2645.174444] Key type .llcrypt registered [ 2646.112950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2646.162392] alg: No test for adler32 (adler32-zlib) [ 2647.537622] Lustre: Lustre: Build Version: 2.17.54_162_gb5ded84 [ 2647.905498] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 2649.775577] Key type lgssc registered [ 2651.740505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2720.273657] Lustre: Mounted lustre-client [ 2726.232802] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2734.852377] Lustre: Unmounted lustre-client [ 2759.392365] Key type lgssc unregistered [ 2759.739195] LNet: 145070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2759.746503] LNetError: 145070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2759.774928] LNet: Removed LNI 192.168.201.53@tcp [ 2761.082108] Key type .llcrypt unregistered [ 2761.083754] Key type ._llcrypt unregistered