[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 475152624 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002295] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007625] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010124] LSM: Security Framework initializing [ 0.011043] Yama: becoming mindful. [ 0.012027] SELinux: Initializing. [ 0.013053] *** VALIDATE selinux *** [ 0.019000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027698] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028247] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032144] *** VALIDATE tmpfs *** [ 0.034231] *** VALIDATE proc *** [ 0.035353] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.038112] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041037] Spectre V2 : User space: Vulnerable [ 0.042006] Speculative Store Bypass: Vulnerable [ 0.045452] debug: unmapping init [mem 0xffffffffaf059000-0xffffffffaf060fff] [ 0.049594] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051860] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052019] ... version: 2 [ 0.053020] ... bit width: 48 [ 0.054013] ... generic registers: 4 [ 0.055009] ... value mask: 0000ffffffffffff [ 0.056011] ... max period: 00007fffffffffff [ 0.057010] ... fixed-purpose events: 3 [ 0.058012] ... event mask: 000000070000000f [ 0.059326] rcu: Hierarchical SRCU implementation. [ 0.062378] smp: Bringing up secondary CPUs ... [ 0.063567] x86: Booting SMP configuration: [ 0.064022] .... node #0, CPUs: #1 #2 #3 [ 0.074116] smp: Brought up 1 node, 4 CPUs [ 0.075973] smpboot: Max logical packages: 1 [ 0.076009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135023] node 0 deferred pages initialised in 57ms [ 0.138257] devtmpfs: initialized [ 0.139182] x86/mm: Memory block size: 128MB [ 0.143764] gcov: version magic: 0x41383552 [ 0.145512] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150160] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152667] pinctrl core: initialized pinctrl subsystem [ 0.154154] [ 0.154688] ************************************************************* [ 0.156013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159013] ** ** [ 0.162017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164014] ** ** [ 0.167015] ** This means that this kernel is built to expose internal ** [ 0.168011] ** IOMMU data structures, which may compromise security on ** [ 0.171016] ** your system. ** [ 0.172013] ** ** [ 0.174027] ** If you see this message and you are not debugging the ** [ 0.176013] ** kernel, report this immediately to your vendor! ** [ 0.177013] ** ** [ 0.179028] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181015] ************************************************************* [ 0.184155] NET: Registered protocol family 16 [ 0.185619] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.187063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.188066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.192788] cpuidle: using governor menu [ 0.195027] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.198547] PCI: Using configuration type 1 for base access [ 0.200123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.205056] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.206015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.208031] cryptd: max_cpu_qlen set to 1000 [ 0.210408] ACPI: Added _OSI(Module Device) [ 0.212012] ACPI: Added _OSI(Processor Device) [ 0.214009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.216010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.221569] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.228119] ACPI: Interpreter enabled [ 0.229037] ACPI: PM: (supports S0 S3 S4 S5) [ 0.229742] ACPI: Using IOAPIC for interrupt routing [ 0.230070] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.232252] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.242412] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.245038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.247016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.250082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.255412] acpiphp: Slot [2] registered [ 0.257086] acpiphp: Slot [5] registered [ 0.258097] acpiphp: Slot [6] registered [ 0.259085] acpiphp: Slot [3] registered [ 0.261106] acpiphp: Slot [4] registered [ 0.262097] acpiphp: Slot [7] registered [ 0.264084] acpiphp: Slot [8] registered [ 0.266086] acpiphp: Slot [9] registered [ 0.267075] acpiphp: Slot [10] registered [ 0.268065] acpiphp: Slot [11] registered [ 0.270084] acpiphp: Slot [12] registered [ 0.271093] acpiphp: Slot [13] registered [ 0.272081] acpiphp: Slot [14] registered [ 0.274062] acpiphp: Slot [15] registered [ 0.274796] acpiphp: Slot [16] registered [ 0.275082] acpiphp: Slot [17] registered [ 0.277093] acpiphp: Slot [18] registered [ 0.278081] acpiphp: Slot [19] registered [ 0.280115] acpiphp: Slot [20] registered [ 0.282081] acpiphp: Slot [21] registered [ 0.283073] acpiphp: Slot [22] registered [ 0.284080] acpiphp: Slot [23] registered [ 0.286089] acpiphp: Slot [24] registered [ 0.288095] acpiphp: Slot [25] registered [ 0.289114] acpiphp: Slot [26] registered [ 0.291100] acpiphp: Slot [27] registered [ 0.292076] acpiphp: Slot [28] registered [ 0.294113] acpiphp: Slot [29] registered [ 0.295070] acpiphp: Slot [30] registered [ 0.296142] acpiphp: Slot [31] registered [ 0.298046] PCI host bridge to bus 0000:00 [ 0.299030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.300015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.302017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.305020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.307025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.310022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.311160] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.313965] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.317275] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.324615] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.329064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.332053] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.334029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.336016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.339564] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.341925] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.344048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.346873] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.352015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.362617] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.367014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.373283] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.385016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.390015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.414013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.433066] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.439016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.446013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.461015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.469795] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.471212] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.472361] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.475352] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.477273] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.483083] iommu: Default domain type: Passthrough [ 0.484614] SCSI subsystem initialized [ 0.486179] ACPI: bus type USB registered [ 0.488113] usbcore: registered new interface driver usbfs [ 0.490096] usbcore: registered new interface driver hub [ 0.491127] usbcore: registered new device driver usb [ 0.493208] pps_core: LinuxPPS API ver. 1 registered [ 0.495017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.499089] PTP clock support registered [ 0.501500] EDAC MC: Ver: 3.0.0 [ 0.503152] PCI: Using ACPI for IRQ routing [ 0.505389] NetLabel: Initializing [ 0.506006] NetLabel: domain hash size = 128 [ 0.506913] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.509101] NetLabel: unlabeled traffic allowed by default [ 0.511334] vgaarb: loaded [ 0.513620] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.516021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.523180] clocksource: Switched to clocksource kvm-clock [ 0.661347] VFS: Disk quotas dquot_6.6.0 [ 0.663246] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.666273] *** VALIDATE ramfs *** [ 0.667857] *** VALIDATE hugetlbfs *** [ 0.669843] pnp: PnP ACPI init [ 0.672842] pnp: PnP ACPI: found 6 devices [ 0.695348] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.697890] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.700179] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.702532] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.704912] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.707014] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.710030] NET: Registered protocol family 2 [ 0.712598] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.717912] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.723508] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.729831] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.735991] TCP: Hash tables configured (established 65536 bind 65536) [ 0.738703] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.741670] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.743973] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.746652] NET: Registered protocol family 1 [ 0.749466] RPC: Registered named UNIX socket transport module. [ 0.752596] RPC: Registered udp transport module. [ 0.755058] RPC: Registered tcp transport module. [ 0.758674] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.761483] NET: Registered protocol family 44 [ 0.764696] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.768581] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.771072] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.774583] PCI: CLS 0 bytes, default 64 [ 0.777302] Unpacking initramfs... [ 2.358382] debug: unmapping init [mem 0xffff989e3cc64000-0xffff989e3ffcffff] [ 2.362317] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.364160] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.367112] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.931718] Initialise system trusted keyrings [ 2.933396] Key type blacklist registered [ 2.934988] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.945132] zbud: loaded [ 2.948247] *** VALIDATE nfs *** [ 2.949556] *** VALIDATE nfs4 *** [ 2.950987] pstore: using deflate compression [ 2.954849] Platform Keyring initialized [ 3.088589] NET: Registered protocol family 38 [ 3.090599] Key type asymmetric registered [ 3.092059] Asymmetric key parser 'x509' registered [ 3.094710] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.097940] io scheduler mq-deadline registered [ 3.099736] io scheduler kyber registered [ 3.101490] io scheduler bfq registered [ 3.103750] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.107322] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.110789] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.114606] ACPI: Power Button [PWRF] [ 3.121560] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.128425] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.139294] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.169304] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.199471] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.204452] Non-volatile memory driver v1.3 [ 3.206087] Linux agpgart interface v0.103 [ 3.236904] virtio_blk virtio1: [vda] 145880 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.239525] vda: detected capacity change from 0 to 74690560 [ 3.274950] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.278085] vdb: detected capacity change from 0 to 1073741824 [ 3.306518] libphy: Fixed MDIO Bus: probed [ 3.319392] usbcore: registered new interface driver usbserial_generic [ 3.321482] usbserial: USB Serial support registered for generic [ 3.323856] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.328737] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.330782] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.333369] mousedev: PS/2 mouse device common for all mice [ 3.336462] rtc_cmos 00:05: RTC can wake from S4 [ 3.343537] rtc_cmos 00:05: registered as rtc0 [ 3.343711] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.345763] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.358494] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.360095] intel_pstate: CPU model not supported [ 3.364048] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.378089] hid: raw HID events driver (C) Jiri Kosina [ 3.380511] usbcore: registered new interface driver usbhid [ 3.382793] usbhid: USB HID core driver [ 3.384994] drop_monitor: Initializing network drop monitor service [ 3.387643] Initializing XFRM netlink socket [ 3.390123] NET: Registered protocol family 10 [ 3.393575] Segment Routing with IPv6 [ 3.395144] NET: Registered protocol family 17 [ 3.397364] mpls_gso: MPLS GSO support [ 3.405009] RAS: Correctable Errors collector initialized. [ 3.406987] AVX version of gcm_enc/dec engaged. [ 3.408825] AES CTR mode by8 optimization enabled [ 3.496448] sched_clock: Marking stable (3496429766, 0)->(4626857712, -1130427946) [ 3.499886] registered taskstats version 1 [ 3.501868] Loading compiled-in X.509 certificates [ 3.504401] zswap: loaded using pool lzo/zbud [ 3.531377] Key type big_key registered [ 3.540813] Key type encrypted registered [ 3.542637] ima: No TPM chip found, activating TPM-bypass! [ 3.544556] ima: Allocated hash algorithm: sha1 [ 3.546218] ima: No architecture policies found [ 3.547794] evm: Initialising EVM extended attributes: [ 3.549283] evm: security.selinux [ 3.550137] evm: security.ima [ 3.550883] evm: security.capability [ 3.551849] evm: HMAC attrs: 0x1 [ 3.553701] rtc_cmos 00:05: setting system clock to 2026-09-02 14:49:53 UTC (1788360593) [ 3.558264] debug: unmapping init [mem 0xffffffffb0003000-0xffffffffb01fffff] [ 3.560163] debug: unmapping init [mem 0xffffffffaed82000-0xffffffffaf058fff] [ 3.565161] Write protecting the kernel read-only data: 28672k [ 3.568877] debug: unmapping init [mem 0xffffffffad403000-0xffffffffad5fffff] [ 3.571326] debug: unmapping init [mem 0xffffffffadd14000-0xffffffffaddfffff] [ 3.601312] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.606771] systemd[1]: Detected virtualization kvm. [ 3.608557] systemd[1]: Detected architecture x86-64. [ 3.610459] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.637298] systemd[1]: No hostname configured. [ 3.638254] systemd[1]: Set hostname to . [ 3.640180] random: systemd: uninitialized urandom read (16 bytes read) [ 3.642023] systemd[1]: Initializing machine ID from random generator. [ 3.682386] random: ln: uninitialized urandom read (6 bytes read) [ 3.804780] random: systemd: uninitialized urandom read (16 bytes read) [ 3.806582] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.810185] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.818385] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.615913] device-mapper: uevent: version 1.0.3 [ 4.619457] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] [ 5.506766] random: fast init done Started Hardware RNG Entropy Gatherer Daemon. [ 5.683424] virtio_net virtio0 ens2: renamed from eth0 [ 6.048222] scsi host0: ata_piix [ 6.070085] scsi host1: ata_piix [ 6.071175] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.072851] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 11.590721] random: crng init done [ 11.596469] random: 7 urandom warning(s) missed due to ratelimiting [ 14.725188] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.003818] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.909583] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.378751] SELinux: Disabled at runtime. [ 22.445581] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.456819] systemd[1]: Detected virtualization kvm. [ 22.459247] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.168396] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.172441] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.219498] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.242068] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.245490] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.253239] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.263460] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ 24.895796] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.549611] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.793390] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.831737] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.294726] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.283622] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 34.237954] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 34.657553] NFS: Registering the id_resolver key type [ 34.659713] Key type id_resolver registered [ 34.661163] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ 36.973720] hrtimer: interrupt took 5183545 ns [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Hostname Service. [ OK ] Started Login Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. Starting Authorization Manager... [ OK ] Started System Logging Service. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg135-client login: [ 104.202640] libcfs: loading out-of-tree module taints kernel. [ 104.305981] Key type ._llcrypt registered [ 104.308938] Key type .llcrypt registered [ 104.704311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 104.724226] alg: No test for adler32 (adler32-zlib) [ 106.048643] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 106.998818] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 108.752685] Key type lgssc registered [ 110.033467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 230.270269] Lustre: Mounted lustre-client [ 235.104887] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 247.868477] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing check_logdir /tmp/testlogs/ [ 253.459478] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing yml_node [ 255.970615] Lustre: lustre-OST0000-osc-ffff989e9200e800: disconnect after 23s idle [ 258.868383] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 261.615329] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 264.064079] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 265.682095] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Sep 2 10:54:14 EDT 2026 [ 282.472172] Lustre: Unmounted lustre-client [ 310.999894] Key type lgssc unregistered [ 311.251127] LNet: 8388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.263186] LNetError: 8388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 311.291976] LNet: Removed LNI 192.168.201.35@tcp [ 312.132163] Key type .llcrypt unregistered [ 312.133632] Key type ._llcrypt unregistered [ 337.094662] Key type ._llcrypt registered [ 337.098663] Key type .llcrypt registered [ 337.329253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 337.361514] alg: No test for adler32 (adler32-zlib) [ 338.280489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 338.327702] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 338.343817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 341.860670] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 342.212616] LNet: 9272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.219504] LNetError: 9272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 342.240661] LNet: Removed LNI 192.168.201.35@tcp [ 343.079495] Key type .llcrypt unregistered [ 343.081116] Key type ._llcrypt unregistered [ 349.528221] Key type ._llcrypt registered [ 349.529685] Key type .llcrypt registered [ 350.882672] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:55:39 (1788360939) [ 351.273048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 351.298565] alg: No test for adler32 (adler32-zlib) [ 352.257251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.350504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.376778] LNet: 9683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.405314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.470710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 352.496987] LNet: 9690:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 358.578738] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:55:47 (1788360947) [ 358.686341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.704154] LNet: 10264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.732189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.815275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 365.355294] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:55:54 (1788360954) [ 365.606101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.637187] LNet: 10856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.709257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.852735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 371.745786] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:56:00 (1788360960) [ 371.884069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.910492] LNet: 11437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.950100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.060752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 378.554650] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:56:07 (1788360967) [ 378.698126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.720835] LNet: 12024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.752131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.800478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 379.076893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 385.569207] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:56:14 (1788360974) [ 385.758562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.828196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.901538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 386.523524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 387.065951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 393.198060] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:56:22 (1788360982) [ 393.272926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.310899] LNet: 13207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.317849] LNet: 13207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 393.342150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.420353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 398.688845] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:56:27 (1788360987) [ 398.838745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.862220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.907611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 398.932706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 399.190888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 399.229530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 399.521527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 399.546039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 399.848898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 399.884422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 400.223801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 400.270030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 400.585107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 400.631560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 400.971916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 401.022503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 401.309632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 401.350339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 401.689182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 401.733199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 402.062158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 402.086923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 402.372233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 402.412799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 402.719133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 402.775910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 403.067537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 403.095401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 408.932166] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:56:37 (1788360997) [ 409.012723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.054976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.164725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 409.200891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 414.436591] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:56:43 (1788361003) [ 414.522700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.550593] LNet: 15060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.558527] LNet: 15060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 414.612767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.708547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 414.781373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 420.602596] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:56:49 (1788361009) [ 420.854662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.912362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.034723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 421.094898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 427.610956] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:56:56 (1788361016) [ 427.697580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.767704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.890353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 427.942900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 434.332715] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:57:03 (1788361023) [ 434.564882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.599682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.669714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 434.708549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 440.584470] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:57:09 (1788361029) [ 440.860799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.916521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.986646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 441.021851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 446.610503] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:57:15 (1788361035) [ 446.734623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.760573] LNet: 17989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 446.768216] LNet: 17989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 446.810825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.971173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 447.350982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 452.951271] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:57:21 (1788361041) [ 453.068763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.105629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.325120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 453.526728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 459.051818] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:57:27 (1788361047) [ 459.142468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.171946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.310314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 459.527795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 465.168466] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:57:33 (1788361053) [ 465.243087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.277145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.745507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.060224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 470.982660] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:57:40 (1788361060) [ 471.150655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.176303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.223828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 471.274584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 475.915158] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:57:44 (1788361064) [ 476.024067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.064336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.142977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.185933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 480.771496] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:57:49 (1788361069) [ 481.047318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.091974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.138241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 486.086686] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:57:55 (1788361075) [ 486.238578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.266368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.315294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 491.459425] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:58:00 (1788361080) [ 491.599670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.658229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.712553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 492.022677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 496.926337] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:58:05 (1788361085) [ 497.091969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.118530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.166619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 497.195205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 502.438657] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:58:11 (1788361091) [ 502.538689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.585411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.684730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 502.732702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 507.740285] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:58:16 (1788361096) [ 507.814984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.843632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.892296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 507.917205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 512.836899] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:58:21 (1788361101) [ 512.910746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.940921] LNet: 25184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.946097] LNet: 25184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 512.997818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.125709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 513.162949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 518.681689] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:58:27 (1788361107) [ 518.932718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.969697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.035355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 519.122981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 519.141895] LNetError: 25790:0:(peer.c:2277:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 519.191658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 519.234162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 524.257146] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:58:33 (1788361113) [ 524.368764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.416688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.476934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.H5yIqumKbq [ 524.542362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.H5yIqumKbq [ 524.597739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.H5yIqumKbq [ 529.739259] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:58:38 (1788361118) [ 529.819066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.860787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.922753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 529.954768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 529.990962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 534.428635] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 10:58:43 (1788361123) [ 534.516682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.548334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.595759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 534.612243] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 535.433736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 536.330517] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 538.119176] Key type lgssc registered [ 538.933434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 541.134303] LNetError: 27550:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.35@tcp99 has no available nets [ 543.011178] Key type lgssc unregistered [ 543.316428] LNetError: 28657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 543.327758] LNet: Removed LNI 192.168.201.35@tcp [ 543.738621] Key type .llcrypt unregistered [ 543.741293] Key type ._llcrypt unregistered [ 547.920138] Key type ._llcrypt registered [ 547.922476] Key type .llcrypt registered [ 548.805656] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 10:58:57 (1788361137) [ 549.074120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 549.094313] alg: No test for adler32 (adler32-zlib) [ 549.982801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.062666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 551.979505] LNet: Message forwarding will be enabled in 2s [ 555.039221] LNet: Message forwarding enabled [ 555.350107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 555.367921] LNet: Message forwarding will stop in 2s to 6s [ 559.804385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 560.451431] LNet: Message forwarding disabled [ 561.537475] LNet: Message forwarding will be enabled in 2s [ 565.600069] LNet: Message forwarding enabled [ 566.355434] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 10:59:15 (1788361155) [ 566.526359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.554520] LNet: 30063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.492913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.559817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 572.044798] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:59:21 (1788361161) [ 572.376743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.414797] LNet: 30644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 572.437253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.478262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 572.511658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 572.534095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 572.552910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 572.586329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 572.607478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 572.643132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 572.681705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 572.726389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 572.774771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 572.820849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 572.863882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 572.919480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 572.964033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 572.990829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 573.013849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 573.035505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 573.056975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 573.084948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 573.109903] LNetError: 30687:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 578.081435] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:59:26 (1788361166) [ 578.199178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.216798] LNet: 31255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 578.236829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.311398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 585.692784] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:59:33 (1788361173) [ 587.414495] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg135-server [ 588.695195] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:59:37 (1788361177) [ 588.874497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.893557] LNet: 32190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.918681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.970681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 589.009587] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 589.750069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[36-40]@tcp [ 589.767966] LNet: 32323:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 589.816449] LNetError: 32194:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.38@tcp is being used as a gateway but routing feature is not turned on [ 589.839975] LNetError: 32194:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.38@tcp has gone from up to down [ 590.825566] LNetError: 32194:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.37@tcp is being used as a gateway but routing feature is not turned on [ 590.835550] LNetError: 32194:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 591.854510] LNetError: 32194:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.36@tcp is being used as a gateway but routing feature is not turned on [ 591.876617] LNetError: 32194:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 593.907256] LNetError: 32194:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.36@tcp is being used as a gateway but routing feature is not turned on [ 593.923851] LNetError: 32194:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 596.371518] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:59:45 (1788361185) [ 597.993167] LNetError: 32194:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.40@tcp is being used as a gateway but routing feature is not turned on [ 598.013604] LNetError: 32194:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 598.128859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg135-server [ 599.830523] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:59:48 (1788361188) [ 599.962819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.992840] LNet: 33254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 600.013115] LNetError: 33254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 600.064204] LNet: Removed LNI 192.168.201.35@tcp [ 600.092929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.156371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 600.182953] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 600.888228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[36-41/2]@tcp [ 600.907582] LNet: 33376:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 600.921170] LNet: 33376:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 600.945839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[36-41/2]@tcp [ 600.961193] LNetError: 33263:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.38@tcp has gone from up to down [ 600.985104] LNetError: 33263:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 606.272188] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:59:55 (1788361195) [ 606.555796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.589216] LNetError: 33968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 606.606764] LNet: Removed LNI 192.168.201.35@tcp [ 606.644334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.733746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 606.789550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.814780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.854817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 606.897281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.919190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.981412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 607.071585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.101401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.171122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 607.232412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.262637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.304613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 607.364954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.392600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.432283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 607.488967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.523415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.599250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 607.667379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.696469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.736915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 607.801685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.833430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.881885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 607.940153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.963268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.021530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 608.059731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.068975] LNet: 34102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.072339] LNet: 34102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 608.082901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.113894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 608.156658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.198579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.244366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 612.972669] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 11:00:01 (1788361201) [ 614.388897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg135-server [ 615.659268] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 11:00:04 (1788361204) [ 617.106796] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg135-server [ 618.394648] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 11:00:07 (1788361207) [ 619.088793] Key type .llcrypt unregistered [ 619.093178] Key type ._llcrypt unregistered [ 624.416043] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 624.540593] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 624.545346] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 624.548350] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 627.092617] Key type ._llcrypt registered [ 627.095054] Key type .llcrypt registered [ 627.228196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 627.237217] alg: No test for adler32 (adler32-zlib) [ 628.122796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.163439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 628.193574] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 628.634575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 628.647463] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 629.917574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 629.939565] LNet: Removed LNI 10.1.2.3@tcp [ 630.035691] LNet: 36103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.040760] LNetError: 36103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.055852] LNet: Removed LNI 192.168.201.35@tcp [ 630.476677] Key type .llcrypt unregistered [ 630.478405] Key type ._llcrypt unregistered [ 639.173591] Key type ._llcrypt registered [ 639.175822] Key type .llcrypt registered [ 640.309076] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 11:00:28 (1788361228) [ 640.772408] Key type .llcrypt unregistered [ 640.775553] Key type ._llcrypt unregistered [ 645.766108] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 645.883691] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 645.886833] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 645.890425] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 648.410740] Key type ._llcrypt registered [ 648.412734] Key type .llcrypt registered [ 648.648272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 648.668527] alg: No test for adler32 (adler32-zlib) [ 649.563932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.617489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 649.636806] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 650.124440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 650.138124] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 651.769459] LNet: 37595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.773706] LNetError: 37595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 651.779373] LNet: Removed LNI 10.1.2.3@tcp [ 651.786730] LNet: Removed LNI 192.168.201.35@tcp [ 652.261612] Key type .llcrypt unregistered [ 652.264230] Key type ._llcrypt unregistered [ 660.366743] Key type ._llcrypt registered [ 660.369286] Key type .llcrypt registered [ 661.507526] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 11:00:50 (1788361250) [ 662.032504] Key type .llcrypt unregistered [ 662.035278] Key type ._llcrypt unregistered [ 666.912312] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 667.135955] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 667.143954] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 667.149359] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 669.616753] Key type ._llcrypt registered [ 669.618431] Key type .llcrypt registered [ 669.756909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 669.765356] alg: No test for adler32 (adler32-zlib) [ 670.632499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.660358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 670.687142] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 672.085620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 672.110128] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 673.672308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 673.695174] LNet: Removed LNI 10.1.2.3@tcp [ 673.707265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 673.730600] LNet: Removed LNI 10.1.2.31@tcp [ 673.745177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 673.778665] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 674.277360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 674.288719] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 674.914422] LNet: 39531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.919187] LNetError: 39531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 674.923515] LNet: Removed LNI 10.1.2.31@tcp [ 674.932913] LNet: Removed LNI 10.1.2.3@tcp [ 675.368878] Key type .llcrypt unregistered [ 675.370784] Key type ._llcrypt unregistered [ 683.976607] Key type ._llcrypt registered [ 683.978911] Key type .llcrypt registered [ 685.202157] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 11:01:13 (1788361273) [ 685.547846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 685.561264] alg: No test for adler32 (adler32-zlib) [ 686.440188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.498972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 686.523203] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 687.042652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.056525] LNet: 40483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.060951] LNetError: 40483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 687.073725] LNet: Removed LNI 192.168.201.35@tcp [ 687.096911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.134883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.35@tcp [ 687.153663] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 687.622785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.35@tcp [ 687.645924] LNet: Removed LNI 192.168.201.35@tcp [ 687.685851] LNet: 40618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.139501] Key type .llcrypt unregistered [ 688.142227] Key type ._llcrypt unregistered [ 696.370462] Key type ._llcrypt registered [ 696.371938] Key type .llcrypt registered [ 697.138471] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 11:01:26 (1788361286) [ 698.361151] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg135-server [ 699.580922] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 11:01:28 (1788361288) [ 700.112545] Key type .llcrypt unregistered [ 700.114229] Key type ._llcrypt unregistered [ 704.989825] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 705.227478] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 705.242081] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 705.254802] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 707.651864] Key type ._llcrypt registered [ 707.656506] Key type .llcrypt registered [ 707.771444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 707.794703] alg: No test for adler32 (adler32-zlib) [ 708.659373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.692498] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 709.159193] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 710.502407] LNet: 42453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.515705] LNetError: 42453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 710.528720] LNet: Removed LNI 192.168.201.35@tcp1 [ 710.541849] LNet: Removed LNI 10.1.2.3@tcp2 [ 711.249702] Key type .llcrypt unregistered [ 711.253505] Key type ._llcrypt unregistered [ 719.974775] Key type ._llcrypt registered [ 719.976939] Key type .llcrypt registered [ 721.048786] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 11:01:49 (1788361309) [ 721.461774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 721.477542] alg: No test for adler32 (adler32-zlib) [ 722.462207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.513373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 722.533924] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 723.331742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.371328] LNet: 43426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.377725] LNetError: 43426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 723.389634] LNet: Removed LNI 192.168.201.35@tcp [ 723.409282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.446820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 723.463148] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 728.864862] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 11:01:57 (1788361317) [ 729.269199] LNet: 44017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.279584] LNetError: 44017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.303238] LNet: Removed LNI 192.168.201.35@tcp [ 729.828723] Key type .llcrypt unregistered [ 729.831910] Key type ._llcrypt unregistered [ 735.363825] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 735.430384] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 735.433344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 735.436472] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 737.895282] Key type ._llcrypt registered [ 737.897253] Key type .llcrypt registered [ 738.035722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 738.055288] alg: No test for adler32 (adler32-zlib) [ 738.944586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.975946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 738.997569] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 739.529281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 739.543259] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 741.071912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.109323] LNet: 44760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.118920] LNetError: 44760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 741.129911] LNet: Removed LNI 10.1.2.3@tcp [ 741.145930] LNet: Removed LNI 192.168.201.35@tcp [ 741.163271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.207271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 741.223310] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 741.226733] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 741.565256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.578767] LNet: Removed LNI 10.1.2.3@tcp [ 741.590408] LNet: Removed LNI 192.168.201.35@tcp [ 741.632411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.682080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 741.697751] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 741.700669] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 742.102759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 742.119353] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 742.787179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 742.806258] LNet: 44949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.810859] LNet: 44949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 742.817810] LNetError: 44949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 742.822091] LNetError: 44949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 742.828559] LNet: Removed LNI 10.1.2.3@tcp [ 742.830383] LNet: Removed LNI 192.168.201.35@tcp [ 742.841653] LNet: Removed LNI 192.168.201.35@tcp2 [ 742.860689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.916268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 742.939303] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 742.942957] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 742.947182] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 748.111089] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 11:02:16 (1788361336) [ 748.243063] LNet: 45538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.257722] LNetError: 45538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 748.266256] LNetError: 45538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 748.273261] LNet: Removed LNI 10.1.2.3@tcp [ 748.278617] LNet: Removed LNI 192.168.201.35@tcp [ 748.294287] LNet: Removed LNI 192.168.201.35@tcp2 [ 748.892061] Key type .llcrypt unregistered [ 748.894068] Key type ._llcrypt unregistered [ 754.288470] Key type ._llcrypt registered [ 754.291694] Key type .llcrypt registered [ 754.515963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 754.530306] alg: No test for adler32 (adler32-zlib) [ 755.446599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.536289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 755.567982] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 756.262837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 756.275506] LNet: 45941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 756.285637] LNetError: 45941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 756.299631] LNet: Removed LNI 192.168.201.35@tcp [ 756.325241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.440796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 756.481416] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 757.155073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.195637] LNet: 45971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.216791] LNetError: 45971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.246258] LNet: Removed LNI 192.168.201.35@tcp [ 757.271032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.380932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 757.411841] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 757.899625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.942576] LNet: Removed LNI 192.168.201.35@tcp [ 757.972652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.050485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 758.079421] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 758.466572] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 758.573640] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 758.580414] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 758.585972] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 761.008441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.029742] LNet: 46079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.038180] LNet: 46079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 761.044558] LNetError: 46079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 761.050627] LNetError: 46079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 761.067919] LNet: Removed LNI 192.168.201.35@tcp [ 761.087325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.148514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 761.166469] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 766.703978] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 11:02:35 (1788361355) [ 767.025634] LNet: 46676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.031234] LNetError: 46676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.053478] LNet: Removed LNI 192.168.201.35@tcp [ 767.656852] Key type .llcrypt unregistered [ 767.658932] Key type ._llcrypt unregistered [ 773.611388] Key type ._llcrypt registered [ 773.613426] Key type .llcrypt registered [ 773.814426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 773.830258] alg: No test for adler32 (adler32-zlib) [ 774.773639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.828800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 774.848447] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 775.774963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.799102] LNet: 47075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.807611] LNetError: 47075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 775.821922] LNet: Removed LNI 192.168.201.35@tcp [ 775.852785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.918672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 775.941399] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 781.609357] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 11:02:50 (1788361370) [ 781.755195] LNet: 47662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.767715] LNetError: 47662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.802322] LNet: Removed LNI 192.168.201.35@tcp [ 782.388513] Key type .llcrypt unregistered [ 782.390085] Key type ._llcrypt unregistered [ 787.810363] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 787.978790] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 787.985823] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 787.991077] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 790.542597] Key type ._llcrypt registered [ 790.544604] Key type .llcrypt registered [ 790.754240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 790.774818] alg: No test for adler32 (adler32-zlib) [ 791.689302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.726869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 791.746046] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 792.327507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.340152] LNet: 48118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.343398] LNetError: 48118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.350812] LNet: Removed LNI 192.168.201.35@tcp [ 792.367964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.414938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 792.433062] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 797.054108] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 11:03:06 (1788361386) [ 797.217792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.240508] LNet: 48716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.247235] LNetError: 48716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.260828] LNet: Removed LNI 192.168.201.35@tcp [ 797.280406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.379870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 802.339054] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 11:03:11 (1788361391) [ 802.528475] LNet: 49298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.029614] Key type .llcrypt unregistered [ 803.033121] Key type ._llcrypt unregistered [ 807.653640] Key type ._llcrypt registered [ 807.655886] Key type .llcrypt registered [ 807.810529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 807.819873] alg: No test for adler32 (adler32-zlib) [ 808.724460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.772447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 808.797588] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 809.723738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.751193] LNet: 49695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.757651] LNetError: 49695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.775273] LNet: Removed LNI 192.168.201.35@tcp [ 809.788976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.832855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 809.849578] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 814.623729] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 11:03:23 (1788361403) [ 814.722474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.749871] LNet: 50282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.759966] LNetError: 50282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.777839] LNet: Removed LNI 192.168.201.35@tcp [ 814.801597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.855340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 819.045526] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 11:03:28 (1788361408) [ 820.591649] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 821.980214] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 11:03:30 (1788361410) [ 823.534594] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 824.992952] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 11:03:33 (1788361413) [ 826.212441] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 827.432525] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 11:03:36 (1788361416) [ 828.951853] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 830.318698] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 11:03:39 (1788361419) [ 831.422393] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 832.997501] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 11:03:41 (1788361421) [ 833.080644] LNet: 52331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.525696] Key type .llcrypt unregistered [ 833.528538] Key type ._llcrypt unregistered [ 837.688627] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 837.782310] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 837.785973] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 837.788613] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 840.286714] Key type ._llcrypt registered [ 840.287919] Key type .llcrypt registered [ 840.383070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 840.393193] alg: No test for adler32 (adler32-zlib) [ 841.289218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.345295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 841.365066] LNet: Added LNI 192.168.201.35@tcp [8/512/0/180] [ 841.370033] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 841.373203] LNet: 52659:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 841.440673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.454227] LNet: 52678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.461588] LNetError: 52678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 841.472307] LNet: Removed LNI 10.1.2.3@tcp [ 841.477917] LNetError: 52678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.487885] LNet: Removed LNI 192.168.201.35@tcp [ 841.510722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.549688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 841.560990] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 841.605248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.617588] LNet: 52702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.621659] LNetError: 52702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.630721] LNet: Removed LNI 192.168.201.35@tcp [ 841.647287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.687213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 841.705195] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 841.712198] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 841.714472] LNet: 52710:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 841.774590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.788316] LNet: 52728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.793185] LNetError: 52728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 841.800371] LNet: Removed LNI 10.1.2.3@tcp [ 841.802249] LNetError: 52728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.811977] LNet: Removed LNI 192.168.201.35@tcp [ 841.832881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.885274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 841.909373] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 841.913892] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 841.915735] LNet: 52736:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 842.031252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.045678] LNet: 52762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.050342] LNetError: 52762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 842.054630] LNet: Removed LNI 10.1.2.3@tcp [ 842.056257] LNetError: 52762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.067947] LNet: Removed LNI 192.168.201.35@tcp [ 842.084244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.142321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 842.160827] LNet: Added LNI 192.168.201.35@tcp [8/256/0/182] [ 842.165272] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 842.228682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.236900] LNet: 52790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.239419] LNetError: 52790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 842.242099] LNet: Removed LNI 10.1.2.3@tcp [ 842.243310] LNetError: 52790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.251900] LNet: Removed LNI 192.168.201.35@tcp [ 842.266160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.318632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 842.337371] LNet: Added LNI 192.168.201.35@tcp [8/256/0/182] [ 842.341218] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 842.342927] LNet: 52798:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 842.397721] LNet: Removed LNI 10.1.2.3@tcp [ 842.405095] LNet: Removed LNI 192.168.201.35@tcp [ 842.847604] Key type .llcrypt unregistered [ 842.848698] Key type ._llcrypt unregistered [ 851.743181] Key type ._llcrypt registered [ 851.744804] Key type .llcrypt registered [ 852.601051] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 11:04:01 (1788361441) [ 852.890119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 852.903711] alg: No test for adler32 (adler32-zlib) [ 853.743417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.786227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 853.805009] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 854.280993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.35 [ 854.295402] LNet: 53773:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 854.301227] LNet: Removed LNI 192.168.201.35@tcp [ 858.326558] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 11:04:07 (1788361447) [ 858.491470] LNet: 54343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.497351] LNetError: 54343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.524398] LNet: Removed LNI 192.168.201.35@tcp [ 858.923476] Key type .llcrypt unregistered [ 858.925959] Key type ._llcrypt unregistered [ 862.626626] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 862.813895] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 865.407872] Key type ._llcrypt registered [ 865.409906] Key type .llcrypt registered [ 865.508344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 865.518653] alg: No test for adler32 (adler32-zlib) [ 866.786562] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 872.174871] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 11:04:20 (1788361460) [ 872.282211] LNet: 55249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.296333] LNetError: 55249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 872.322237] LNet: Removed LNI 10.1.2.3@tcp [ 872.821413] Key type .llcrypt unregistered [ 872.822697] Key type ._llcrypt unregistered [ 877.974976] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 878.196615] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 878.199822] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 878.209947] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 880.922964] Key type ._llcrypt registered [ 880.924803] Key type .llcrypt registered [ 881.102384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 881.137071] alg: No test for adler32 (adler32-zlib) [ 882.058397] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 886.671205] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 11:04:35 (1788361475) [ 887.620196] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 888.739572] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 11:04:37 (1788361477) [ 888.917517] LNet: 56441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.921992] LNetError: 56441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 888.937264] LNet: Removed LNI 10.1.2.3@tcp [ 889.392568] Key type .llcrypt unregistered [ 889.395774] Key type ._llcrypt unregistered [ 894.097239] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 894.264977] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 894.266716] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 894.268681] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 896.961301] Key type ._llcrypt registered [ 896.964639] Key type .llcrypt registered [ 897.133045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 897.150455] alg: No test for adler32 (adler32-zlib) [ 898.114875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.213974] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 904.721325] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 11:04:53 (1788361493) [ 905.073177] LNet: 57672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.078363] LNetError: 57672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 905.093581] LNet: Removed LNI 10.1.2.3@tcp [ 905.653419] Key type .llcrypt unregistered [ 905.654981] Key type ._llcrypt unregistered [ 909.995754] Key type ._llcrypt registered [ 909.997724] Key type .llcrypt registered [ 910.123550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 910.135254] alg: No test for adler32 (adler32-zlib) [ 910.992566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 911.015686] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 916.901443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 917.424208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_interrupt -r 1 [ 917.455706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_interrupt -r 1 [ 917.491425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 917.515357] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 917.661378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 917.675258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 917.703172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 917.730370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 917.749589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 917.771914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 917.795835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 917.831225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 917.849965] LNet: 58110:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 917.865659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 917.904430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 917.922667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 917.954186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 917.990312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 918.019177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 918.039891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 918.350040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_dropped -r 1 [ 918.377360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_dropped -r 1 [ 918.402220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 918.415320] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 918.582440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 918.605726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 918.628631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 918.648540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 918.670747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 918.705116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 918.725785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 918.748789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 918.768560] LNet: 58219:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 918.783182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 918.809590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 918.831845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 918.851921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 918.877134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 918.898249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 918.923710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 919.273953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_aborted -r 1 [ 919.317468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_aborted -r 1 [ 919.344866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 919.506582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 919.530441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 919.555249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 919.573916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 919.597517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 919.620385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 919.640511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 919.666110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 919.683891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 919.699955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 919.722811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 919.747971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 919.769985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 919.804772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 919.829915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 920.305600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_no_route -r 1 [ 920.338824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_no_route -r 1 [ 920.370828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 920.391597] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 920.399812] LNet: Skipped 1 previous similar message [ 920.566544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 920.599849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 920.625185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 920.647435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 920.683160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 920.705176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 920.721567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 920.743483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 920.759499] LNet: 58437:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 920.766715] LNet: 58437:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 920.782640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 920.814698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 920.851117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 920.882711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 920.913143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 920.943322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 920.965877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 921.369926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_timeout -r 1 [ 921.392214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_timeout -r 1 [ 921.419647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 921.607951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 921.636383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 921.657802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 921.685594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 921.707950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 921.726589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 921.762744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 921.792881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 921.813309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 921.836727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 921.866717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 921.894362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 921.920476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 921.958618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 921.984659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 922.380943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_error -r 1 [ 922.412645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_error -r 1 [ 922.438699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 922.455308] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 922.459692] LNet: Skipped 1 previous similar message [ 922.623351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 922.658582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 922.693617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 922.712342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 922.742985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 922.763863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 922.795279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 922.819267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 922.830348] LNet: 58656:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 922.834835] LNet: 58656:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 922.847920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 922.870977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 922.899893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 922.931709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 922.953570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 922.980868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 923.005835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 923.171915] LNet: 58697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.176274] LNetError: 58697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.191542] LNet: Removed LNI 192.168.201.35@tcp [ 923.468768] Key type .llcrypt unregistered [ 923.473050] Key type ._llcrypt unregistered [ 936.746952] Key type ._llcrypt registered [ 936.749439] Key type .llcrypt registered [ 937.847681] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 11:05:26 (1788361526) [ 938.529620] Key type .llcrypt unregistered [ 938.530844] Key type ._llcrypt unregistered [ 942.588439] Key type ._llcrypt registered [ 942.589570] Key type .llcrypt registered [ 942.686198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 942.693532] alg: No test for adler32 (adler32-zlib) [ 943.558889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 943.582868] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 948.761478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 956.101496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.145737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 956.182475] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 957.251455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_interrupt -r 1 [ 957.302881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_interrupt -r 1 [ 957.350776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_interrupt -r 1 [ 957.381373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_interrupt -r 1 [ 957.404845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_interrupt -r 1 [ 957.439978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_interrupt -r 1 [ 957.469891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_interrupt -r 1 [ 957.511883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_interrupt -r 1 [ 957.553993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 957.568232] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 957.574971] LNet: 1 local NIs in recovery (showing 1): 192.168.201.35@tcp [ 957.759898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 957.783629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 957.814445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 957.837805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 957.867291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 957.887638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 957.908869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 957.932717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 957.945343] LNet: 60162:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 957.957322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 957.976581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 957.996276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 958.018949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 958.042377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 958.065520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 958.093207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 958.496971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_dropped -r 1 [ 958.538966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_dropped -r 1 [ 958.571893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_dropped -r 1 [ 958.602230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_dropped -r 1 [ 958.628793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_dropped -r 1 [ 958.660699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_dropped -r 1 [ 958.684450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_dropped -r 1 [ 958.711506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_dropped -r 1 [ 958.740405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 958.757796] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 958.762227] LNet: Skipped 2 previous similar messages [ 958.927286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 958.957497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 958.984349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 959.008709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 959.031978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 959.055581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 959.085471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 959.121378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 959.141939] LNet: 60284:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 959.158447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 959.181157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 959.213580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 959.243913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 959.277879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 959.304885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 959.332511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 959.657895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_aborted -r 1 [ 959.682842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_aborted -r 1 [ 959.708669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_aborted -r 1 [ 959.734983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_aborted -r 1 [ 959.760353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_aborted -r 1 [ 959.788658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_aborted -r 1 [ 959.817408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_aborted -r 1 [ 959.845650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_aborted -r 1 [ 959.880342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 959.899624] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 959.903202] LNet: Skipped 2 previous similar messages [ 960.057121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 960.076107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 960.101303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 960.130637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 960.156535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 960.190667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 960.216386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 960.242798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 960.257414] LNet: 60405:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 960.274917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 960.299843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 960.322646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 960.342309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 960.365542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 960.386443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 960.406121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 960.769850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_no_route -r 1 [ 960.794947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_no_route -r 1 [ 960.827839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_no_route -r 1 [ 960.853990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_no_route -r 1 [ 960.886945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_no_route -r 1 [ 960.920947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_no_route -r 1 [ 960.951926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_no_route -r 1 [ 960.973901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_no_route -r 1 [ 960.990566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 961.146521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 961.166800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 961.187710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 961.212612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 961.234927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 961.257563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 961.278973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 961.302623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 961.324223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 961.345897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 961.369433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 961.401666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 961.429797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 961.458708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 961.493490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 961.890526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_timeout -r 1 [ 961.916734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_timeout -r 1 [ 961.939348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_timeout -r 1 [ 961.967979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_timeout -r 1 [ 961.993574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_timeout -r 1 [ 962.025255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_timeout -r 1 [ 962.044590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_timeout -r 1 [ 962.069878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_timeout -r 1 [ 962.099705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 962.114389] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 962.117068] LNet: Skipped 5 previous similar messages [ 962.271441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 962.328876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 962.363895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 962.399468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 962.435458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 962.480218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 962.525958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 962.574328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 962.593788] LNet: 60648:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 962.601438] LNet: 60648:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 962.627945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 962.663519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 962.694027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 962.723250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 962.750400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 962.779046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 962.817750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 963.201402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_error -r 1 [ 963.235518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_error -r 1 [ 963.264503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_error -r 1 [ 963.293534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_error -r 1 [ 963.320626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_error -r 1 [ 963.345844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_error -r 1 [ 963.371338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_error -r 1 [ 963.397437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_error -r 1 [ 963.422996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 963.616274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 963.638985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 963.669401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 963.698318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 963.714213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 963.732831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 963.757133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 963.782855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 963.803702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 963.819767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 963.842739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 963.864913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 963.894201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 963.920367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 963.955429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 967.440296] LNet: 60844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.446417] LNetError: 60844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.452867] LNet: Removed LNI 192.168.201.35@tcp [ 967.467055] LNet: Removed LNI 192.168.201.35@tcp1 [ 967.714217] Key type .llcrypt unregistered [ 967.716215] Key type ._llcrypt unregistered [ 978.402324] Key type ._llcrypt registered [ 978.408250] Key type .llcrypt registered [ 979.196579] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 11:06:08 (1788361568) [ 979.692426] Key type .llcrypt unregistered [ 979.694254] Key type ._llcrypt unregistered [ 983.757698] Key type ._llcrypt registered [ 983.759715] Key type .llcrypt registered [ 983.892598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 983.904808] alg: No test for adler32 (adler32-zlib) [ 984.748828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 984.777065] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 989.776454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 990.273816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e remote_dropped -r 1 [ 990.321172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e remote_dropped -r 1 [ 990.374537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 990.398472] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 990.545853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 990.569673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 990.590371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 990.618731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 990.645396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 990.674677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 990.696613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 990.722629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 990.737918] LNet: 62093:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 990.757495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 990.787848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 990.812302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 990.841742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 990.872301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 990.903854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 990.930315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 991.270889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e remote_error -r 1 [ 991.304174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e remote_error -r 1 [ 991.328346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 991.341124] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 991.462772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 991.479130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 991.511405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 991.536877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 991.554835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 991.572613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 991.593097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 991.613888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 991.628267] LNet: 62202:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 991.643619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 991.674796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 991.696445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 991.720696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 991.744412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 991.769612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 991.790965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 992.120430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e remote_timeout -r 1 [ 992.138953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e remote_timeout -r 1 [ 992.166109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 992.353516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 992.375293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 992.407277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 992.432541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 992.467036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 992.505134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 992.526472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 992.560201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 992.590565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 992.627620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 992.660659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 992.690749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 992.713233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 992.743782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 992.766520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 992.928343] LNet: 62352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.931535] LNetError: 62352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.942586] LNet: Removed LNI 192.168.201.35@tcp [ 993.172804] Key type .llcrypt unregistered [ 993.174672] Key type ._llcrypt unregistered [ 1003.874820] Key type ._llcrypt registered [ 1003.877758] Key type .llcrypt registered [ 1004.748108] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 11:06:33 (1788361593) [ 1005.237293] Key type .llcrypt unregistered [ 1005.240074] Key type ._llcrypt unregistered [ 1009.310262] Key type ._llcrypt registered [ 1009.313179] Key type .llcrypt registered [ 1009.573452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.603937] alg: No test for adler32 (adler32-zlib) [ 1010.515845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1010.538757] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1016.574115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1023.040135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.073436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1023.088770] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1024.041806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e remote_dropped -r 1 [ 1024.068330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e remote_dropped -r 1 [ 1024.094896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e remote_dropped -r 1 [ 1024.134308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e remote_dropped -r 1 [ 1024.160167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e remote_dropped -r 1 [ 1024.185778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e remote_dropped -r 1 [ 1024.214159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e remote_dropped -r 1 [ 1024.244406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e remote_dropped -r 1 [ 1024.274211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1024.297290] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1024.303074] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.135@tcp [ 1024.478267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1024.500380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1024.532766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1024.556664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1024.580876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1024.611405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1024.649079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1024.673367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1024.691973] LNet: 63818:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1024.705081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1024.724233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1024.746488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1024.763567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1024.785117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1024.807739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1024.830633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1024.857943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1025.168894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e remote_error -r 1 [ 1025.192610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e remote_error -r 1 [ 1025.214573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e remote_error -r 1 [ 1025.242990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e remote_error -r 1 [ 1025.272123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e remote_error -r 1 [ 1025.318203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e remote_error -r 1 [ 1025.363980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e remote_error -r 1 [ 1025.395458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e remote_error -r 1 [ 1025.439971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1025.462168] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1025.466929] LNet: Skipped 2 previous similar messages [ 1025.622464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1025.643903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1025.665782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1025.686561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1025.712158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1025.734921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1025.759948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1025.783784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1025.794689] LNet: 63941:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1025.805317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1025.833990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1025.870563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1025.893155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1025.917949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1025.946389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1025.973231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1026.000343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1026.362765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e remote_timeout -r 1 [ 1026.391505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e remote_timeout -r 1 [ 1026.416803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e remote_timeout -r 1 [ 1026.443405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e remote_timeout -r 1 [ 1026.470944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e remote_timeout -r 1 [ 1026.503670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e remote_timeout -r 1 [ 1026.527652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e remote_timeout -r 1 [ 1026.558707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e remote_timeout -r 1 [ 1026.593235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1026.611399] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1026.763172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1026.780528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1026.799231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1026.820214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1026.848558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1026.873180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1026.896147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1026.934826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1026.948596] LNet: 64064:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1026.958171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1026.978378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1026.996662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1027.020470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1027.050496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1027.078338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1027.105646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1027.145511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1031.229295] LNet: 64142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.236217] LNetError: 64142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1031.244303] LNet: Removed LNI 192.168.201.35@tcp [ 1031.263511] LNet: Removed LNI 192.168.201.35@tcp1 [ 1031.593717] Key type .llcrypt unregistered [ 1031.595448] Key type ._llcrypt unregistered [ 1042.317162] Key type ._llcrypt registered [ 1042.318762] Key type .llcrypt registered [ 1043.107576] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 11:07:12 (1788361632) [ 1044.031816] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1045.051894] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 11:07:14 (1788361634) [ 1045.784583] Key type .llcrypt unregistered [ 1045.786173] Key type ._llcrypt unregistered [ 1050.081725] Key type ._llcrypt registered [ 1050.084178] Key type .llcrypt registered [ 1050.196537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1050.207575] alg: No test for adler32 (adler32-zlib) [ 1051.082466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1051.101394] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1056.773185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1057.397683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e network_timeout -r 1 [ 1057.424764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e network_timeout -r 1 [ 1057.448398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1057.469171] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1057.651929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1057.679599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1057.707755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1057.731777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1057.759954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1057.778304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1057.799971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1057.821838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1057.835715] LNet: 65685:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1057.849464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1057.869400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1057.890632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1057.909240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1057.929524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1057.948473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1058.119225] LNet: 65724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.123499] LNetError: 65724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.136899] LNet: Removed LNI 192.168.201.35@tcp [ 1058.442719] Key type .llcrypt unregistered [ 1058.444865] Key type ._llcrypt unregistered [ 1071.539966] Key type ._llcrypt registered [ 1071.541590] Key type .llcrypt registered [ 1071.695567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1071.712528] alg: No test for adler32 (adler32-zlib) [ 1072.588430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1072.618259] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1078.377723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1085.316433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1085.375230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1085.394405] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1086.583389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e network_timeout -r 1 [ 1086.615841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e network_timeout -r 1 [ 1086.648864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e network_timeout -r 1 [ 1086.687809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e network_timeout -r 1 [ 1086.719866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e network_timeout -r 1 [ 1086.741929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e network_timeout -r 1 [ 1086.768663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e network_timeout -r 1 [ 1086.795974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e network_timeout -r 1 [ 1086.825657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1086.840319] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1086.944335] LNet: 1 local NIs in recovery (showing 1): 192.168.201.35@tcp [ 1086.953590] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.135@tcp [ 1087.028388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1087.048649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1087.070383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1087.094374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1087.113853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1087.137955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1087.164818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1087.191977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1087.209349] LNet: 66603:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1087.229539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1087.266880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1087.288770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1087.312740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1087.328187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1087.341294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1090.774726] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1091.091061] LNet: 66677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.097927] LNetError: 66677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1091.104145] LNet: Removed LNI 192.168.201.35@tcp [ 1091.120068] LNet: Removed LNI 192.168.201.35@tcp1 [ 1091.528220] Key type .llcrypt unregistered [ 1091.532772] Key type ._llcrypt unregistered [ 1103.932953] Key type ._llcrypt registered [ 1103.936629] Key type .llcrypt registered [ 1104.787409] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 11:08:13 (1788361693) [ 1105.110322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1105.121699] alg: No test for adler32 (adler32-zlib) [ 1106.031194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1106.079502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1106.095983] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1106.597141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1106.610169] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1107.214555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1107.255734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1107.286305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1107.317692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1107.341060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1107.362494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1107.387519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.35@tcp [ 1107.404953] LNetError: 67776:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.35@tcp to recovery [ 1108.127877] LNet: 1 local NIs in recovery (showing 1): 192.168.201.35@tcp [ 1108.128189] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1110.175626] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1110.769125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1114.271764] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1115.276770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1115.307680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1115.385260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1115.411236] LNet: 67893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.418697] LNetError: 67893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.426158] LNet: Removed LNI 192.168.201.35@tcp [ 1115.436671] LNet: Removed LNI 192.168.201.35@tcp1 [ 1115.464899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1115.534261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1115.553206] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1115.988596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1115.999743] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1116.542512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1116.574507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.627751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1116.686965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1116.722384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1116.749696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1116.791602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1116.817958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.35@tcp [ 1116.833251] LNetError: 68166:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.35@tcp to recovery [ 1117.536971] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1120.218144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1123.679418] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1123.688046] LNet: Skipped 1 previous similar message [ 1127.913832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1127.940752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1127.960645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1132.735690] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 11:08:41 (1788361721) [ 1132.795074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1132.809522] LNet: 68877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1132.817320] LNetError: 68877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1132.824530] LNetError: 68877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1132.830584] LNet: Removed LNI 192.168.201.35@tcp [ 1132.857413] LNet: Removed LNI 192.168.201.35@tcp1 [ 1132.867824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1132.911796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1132.935945] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1133.552217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1133.569944] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1134.253407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1134.290405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1134.307961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1134.327054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1134.341311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1134.362636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1134.395512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.35@tcp --health 0 [ 1134.469365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1134.943157] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.35@tcp [ 1135.967446] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1135.974247] LNet: Skipped 1 previous similar message [ 1150.482434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1150.570639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1150.605864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1150.627887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.35@tcp --health 500 [ 1150.693433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1157.111908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1157.159767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1157.173266] LNet: 69383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1157.178872] LNetError: 69383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1157.183714] LNetError: 69383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1157.189329] LNet: Removed LNI 192.168.201.35@tcp [ 1157.201260] LNet: Removed LNI 192.168.201.35@tcp1 [ 1157.223328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1157.260717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1157.281219] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1157.765345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1157.778392] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1158.301141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1158.323490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1158.369818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1158.392454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1158.417329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1158.453452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1158.476491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1158.517460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.35@tcp --health 0 [ 1159.330245] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1159.335908] LNet: Skipped 3 previous similar messages [ 1159.756226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1170.700264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1170.736868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1170.760584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.604872] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 11:09:24 (1788361764) [ 1175.888562] LNet: 70439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.892875] LNetError: 70439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.896743] LNetError: 70439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1175.901633] LNet: Removed LNI 192.168.201.35@tcp [ 1175.913197] LNet: Removed LNI 192.168.201.35@tcp1 [ 1176.408925] Key type .llcrypt unregistered [ 1176.415896] Key type ._llcrypt unregistered [ 1181.163241] Key type ._llcrypt registered [ 1181.166151] Key type .llcrypt registered [ 1181.320327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1181.328671] alg: No test for adler32 (adler32-zlib) [ 1182.274236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1182.305885] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1189.228215] Lustre: DEBUG MARKER: Initial discovery [ 1189.325130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.135@tcp [ 1190.869323] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1190.910875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1190.937721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.135@tcp [ 1190.953047] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1190.976025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1191.008138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.135@tcp -r 1 -m GET -l 3 [ 1191.048899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.135@tcp -r 1 -m GET -e local_error [ 1191.089300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.135@tcp -r 1 -m PUT -l 6 [ 1191.519764] LNet: 1 local NIs in recovery (showing 1): 192.168.201.35@tcp [ 1192.119500] Lustre: DEBUG MARKER: Force oleg135-client.virtnet to discover 192.168.201.135@tcp (in background) [ 1192.279806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.135@tcp [ 1193.129138] Lustre: DEBUG MARKER: Force oleg135-server to discover 192.168.201.35@tcp [ 1195.006194] Lustre: DEBUG MARKER: Wait for 71202 [ 1195.367304] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1196.581656] Lustre: DEBUG MARKER: Finished wait on 71202 [ 1196.622856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1196.657882] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1196.665142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1196.901313] LNet: 71650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1196.908155] LNetError: 71650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1196.922745] LNet: Removed LNI 192.168.201.35@tcp [ 1197.223589] Key type .llcrypt unregistered [ 1197.228217] Key type ._llcrypt unregistered [ 1208.755681] Key type ._llcrypt registered [ 1208.757808] Key type .llcrypt registered [ 1209.567585] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 11:09:58 (1788361798) [ 1209.943174] Key type .llcrypt unregistered [ 1209.945618] Key type ._llcrypt unregistered [ 1214.581212] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1214.680062] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1214.682492] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1214.696845] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1217.252574] Key type ._llcrypt registered [ 1217.257694] Key type .llcrypt registered [ 1217.440717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1217.453160] alg: No test for adler32 (adler32-zlib) [ 1218.376787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1218.434469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1218.459685] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1218.944935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1218.959128] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1224.369387] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 11:10:13 (1788361813) [ 1224.473576] LNet: 73724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1224.479788] LNetError: 73724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1224.484470] LNet: Removed LNI 10.1.2.3@tcp [ 1224.495729] LNet: Removed LNI 192.168.201.35@tcp [ 1224.930377] Key type .llcrypt unregistered [ 1224.932387] Key type ._llcrypt unregistered [ 1229.868778] Key type ._llcrypt registered [ 1229.870672] Key type .llcrypt registered [ 1229.999530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1230.015409] alg: No test for adler32 (adler32-zlib) [ 1230.893842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1230.930273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1230.953204] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1231.466937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1231.482469] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1232.159940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.35@tcp [ 1233.247264] LNetError: Unexpected error -22 connecting to 192.168.201.35@tcp at host 192.168.201.35:988 [ 1236.838687] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 11:10:25 (1788361825) [ 1236.896276] LNet: 74848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1236.903448] LNetError: 74848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1236.907930] LNet: Removed LNI 10.1.2.3@tcp [ 1236.916319] LNet: Removed LNI 192.168.201.35@tcp [ 1237.310335] Key type .llcrypt unregistered [ 1237.311678] Key type ._llcrypt unregistered [ 1241.799501] Key type ._llcrypt registered [ 1241.802588] Key type .llcrypt registered [ 1241.955146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1241.974592] alg: No test for adler32 (adler32-zlib) [ 1242.880969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1242.925664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1242.944821] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1243.453873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1243.474289] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 1244.169453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.35@tcp1 --nid 192.168.201.35@tcp2 [ 1246.966537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp1 192.168.201.35@tcp2 [ 1246.978498] LNetError: 75692:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.35@tcp1 to 192.168.201.35@tcp2 - Route table may be misconfigured [ 1246.997593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp2 192.168.201.35@tcp1 [ 1251.238701] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 11:10:40 (1788361840) [ 1251.441968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1251.467892] LNet: 76265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.474783] LNetError: 76265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1251.479489] LNet: Removed LNI 192.168.201.35@tcp1 [ 1252.524694] LNet: Removed LNI 192.168.201.35@tcp2 [ 1252.561645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1252.620779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1252.645236] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1253.118225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1253.131648] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1253.696923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1253.736422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp 192.168.201.35@tcp [ 1253.793819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp1 192.168.201.35@tcp1 [ 1253.835491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.35@tcp -d 192.168.201.35@tcp -e network_timeout [ 1253.873117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -e network_timeout [ 1253.904442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -e network_timeout [ 1253.939273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -e network_timeout [ 1253.984517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1254.020421] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1254.045924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1254.109664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1254.183977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1254.623105] LNet: 1 local NIs in recovery (showing 1): 192.168.201.35@tcp [ 1258.832980] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 11:10:47 (1788361847) [ 1258.941391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1258.971101] LNet: 77125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.975296] LNetError: 77125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1258.979958] LNetError: 77125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1258.986839] LNet: Removed LNI 192.168.201.35@tcp [ 1260.005383] LNet: Removed LNI 192.168.201.35@tcp1 [ 1260.037994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1260.082633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1260.212831] LNet: 77164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1260.454782] Key type .llcrypt unregistered [ 1260.458151] Key type ._llcrypt unregistered [ 1268.790726] Key type ._llcrypt registered [ 1268.793464] Key type .llcrypt registered [ 1269.669323] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 11:10:58 (1788361858) [ 1270.122862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1270.141697] alg: No test for adler32 (adler32-zlib) [ 1270.998255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1272.058374] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1273.356538] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 11:11:02 (1788361862) [ 1273.580749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1273.598188] LNet: 78261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.615632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1273.664664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1273.689747] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1274.268172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1274.286918] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1275.067671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1275.111165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp1 [ 1275.158292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp1 [ 1279.721533] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 11:11:08 (1788361868) [ 1280.775468] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg135-server [ 1281.835874] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 11:11:10 (1788361870) [ 1283.243787] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg135-server [ 1284.264954] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 11:11:13 (1788361873) [ 1285.663266] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg135-server [ 1286.739555] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 11:11:15 (1788361875) [ 1287.925380] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg135-server [ 1288.960441] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 11:11:18 (1788361878) [ 1290.120508] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg135-server [ 1291.152711] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 11:11:20 (1788361880) [ 1292.254367] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg135-server [ 1293.210459] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 11:11:22 (1788361882) [ 1294.252702] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg135-server [ 1295.284820] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 11:11:24 (1788361884) [ 1296.343508] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg135-server [ 1297.283964] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 11:11:26 (1788361886) [ 1298.383913] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg135-server [ 1299.526812] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 11:11:28 (1788361888) [ 1299.691567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1299.704363] LNet: 82137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1299.708713] LNetError: 82137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1300.767087] LNet: Removed LNI 192.168.201.35@tcp [ 1300.771480] LNetError: 82137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1300.790826] LNet: Removed LNI 192.168.201.35@tcp1 [ 1300.818114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1300.853153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1300.868249] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1301.338836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1301.384689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1301.484592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1301.497151] LNet: 82284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1302.575835] LNet: Removed LNI 192.168.201.35@tcp [ 1302.593885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1302.637790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1302.660183] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1303.150417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1303.271174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1303.289487] LNetError: 82429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1303.293601] LNetError: 82429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1304.358973] LNet: Removed LNI 192.168.201.35@tcp [ 1304.389289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1304.419416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1304.436450] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1304.958680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1305.020796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1305.119294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1305.129878] LNet: 82577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1305.134379] LNet: 82577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1306.154035] LNet: Removed LNI 192.168.201.35@tcp [ 1306.168805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1306.211380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1306.229338] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1306.727941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1306.862721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1306.874688] LNetError: 82722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.877289] LNetError: 82722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1307.942345] LNet: Removed LNI 192.168.201.35@tcp [ 1307.959437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1308.009498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1308.032693] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1308.446377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1308.498915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1309.754724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1309.776281] LNet: 82878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.780521] LNet: 82878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1310.824671] LNet: Removed LNI 192.168.201.35@tcp [ 1310.840819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1310.885406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1310.926455] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1311.617806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1311.816426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1311.842305] LNetError: 83024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1311.846285] LNetError: 83024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1312.871156] LNet: Removed LNI 192.168.201.35@tcp [ 1312.896802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1312.938965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1312.955900] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1313.379196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1313.430815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1313.531239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1314.604187] LNet: Removed LNI 192.168.201.35@tcp [ 1314.628778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1314.681557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1314.699308] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1315.252158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1315.364957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1316.391762] LNet: Removed LNI 192.168.201.35@tcp [ 1316.404864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1316.431889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1316.444830] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1316.917569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1316.978053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1317.170042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1318.247477] LNet: Removed LNI 192.168.201.35@tcp [ 1318.268346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1318.320271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1318.348266] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1318.922653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1319.066301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1319.099614] LNet: 83609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1319.111602] LNet: 83609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1320.167837] LNet: Removed LNI 192.168.201.35@tcp [ 1320.195374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1320.251139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1320.275236] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1320.720676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1320.782189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1320.931774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1320.952598] LNetError: 83757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1320.964217] LNetError: 83757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1320.991716] LNet: Removed LNI 192.168.201.35@tcp [ 1321.009559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1321.058312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1321.079984] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1321.630358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1321.798918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1322.864665] LNet: Removed LNI 192.168.201.35@tcp [ 1322.886958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1322.924747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1322.938533] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1323.489225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1323.545218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1323.579281] LNet: Removed LNI 192.168.201.35@tcp [ 1323.595611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1323.633479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1323.654340] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1324.208601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1324.274846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1324.300925] LNet: Removed LNI 192.168.201.35@tcp [ 1324.324570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.376703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1324.444306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1324.470590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.519232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1324.576397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1324.600164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.634369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1324.670856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1324.696468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.729824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1324.744677] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1325.217095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1325.241258] LNet: Removed LNI 192.168.201.35@tcp [ 1325.253699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1325.304807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1329.525537] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 11:11:58 (1788361918) [ 1329.633155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1329.672937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1329.707935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1329.723138] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/180] [ 1330.439971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1330.466690] LNet: Removed LNI 192.168.201.35@tcp231 [ 1330.480571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1330.527311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1330.557185] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/180] [ 1330.561291] LNet: 85068:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1330.899283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1330.922782] LNet: Removed LNI 192.168.201.35@tcp231 [ 1330.935280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1330.958529] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/180] [ 1331.614174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1331.640627] LNet: Removed LNI 192.168.201.35@tcp231 [ 1331.658845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1331.726223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1331.752690] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/0] [ 1331.759638] LNet: 85221:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1332.163995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1332.193520] LNet: Removed LNI 192.168.201.35@tcp231 [ 1332.206150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1332.223524] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/0] [ 1332.912957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1332.942410] LNet: Removed LNI 192.168.201.35@tcp231 [ 1332.956833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1333.004454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1333.023710] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/60] [ 1333.028977] LNet: 85372:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1333.265283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1333.293771] LNet: Removed LNI 192.168.201.35@tcp231 [ 1333.303955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1333.326146] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/60] [ 1334.106560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1334.135947] LNet: Removed LNI 192.168.201.35@tcp231 [ 1334.157849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1334.223381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1334.248720] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/180] [ 1334.513810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1334.543809] LNet: Removed LNI 192.168.201.35@tcp231 [ 1334.561767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1334.582502] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/180] [ 1339.422304] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 11:12:08 (1788361928) [ 1339.594186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1339.606797] LNet: 86233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1339.610937] LNet: 86233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 1339.615710] LNetError: 86233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1339.619758] LNetError: 86233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1339.632798] LNet: Removed LNI 192.168.201.35@tcp231 [ 1339.642517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1339.675226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1339.688532] LNet: Added LNI 192.168.201.35@tcp232 [8/256/0/180] [ 1340.170735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1340.204234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1340.231466] LNet: Removed LNI 192.168.201.35@tcp232 [ 1340.246018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1340.270642] LNet: Added LNI 192.168.201.35@tcp232 [8/256/0/180] [ 1340.788896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1340.819618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1340.862500] LNet: Removed LNI 192.168.201.35@tcp232 [ 1340.871514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1340.886113] LNet: Added LNI 192.168.201.35@tcp232 [8/256/0/180] [ 1341.329236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1341.355489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1341.378506] LNet: Removed LNI 192.168.201.35@tcp232 [ 1341.390364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1341.406651] LNet: Added LNI 192.168.201.35@tcp232 [8/256/0/180] [ 1341.941485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1341.967969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1341.989435] LNet: Removed LNI 192.168.201.35@tcp232 [ 1341.999387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1345.789838] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 11:12:14 (1788361934) [ 1346.488592] Key type .llcrypt unregistered [ 1346.494360] Key type ._llcrypt unregistered [ 1349.916993] Key type ._llcrypt registered [ 1349.919288] Key type .llcrypt registered [ 1350.071305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1350.081161] alg: No test for adler32 (adler32-zlib) [ 1350.975857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1351.013393] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1355.859216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1360.989640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1361.026575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1361.047544] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1361.961034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_interrupt -r 2 [ 1361.991267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_interrupt -r 2 [ 1362.020472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_interrupt -r 2 [ 1362.052252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_interrupt -r 2 [ 1362.083650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_interrupt -r 2 [ 1362.124319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_interrupt -r 2 [ 1362.159950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_interrupt -r 2 [ 1362.196754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_interrupt -r 2 [ 1362.224892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp 192.168.201.135@tcp [ 1362.243633] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1362.247886] LNet: 1 local NIs in recovery (showing 1): 192.168.201.35@tcp [ 1362.459437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1362.488763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1362.514716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1362.537352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1362.556186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1362.579522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1362.601492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1362.622828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1362.637248] LNet: 88022:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1362.652657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1362.684844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1362.727482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1362.757757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1362.782715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1362.804520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1363.287591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_dropped -r 2 [ 1363.315474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_dropped -r 2 [ 1363.347591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_dropped -r 2 [ 1363.386106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_dropped -r 2 [ 1363.410476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_dropped -r 2 [ 1363.440340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_dropped -r 2 [ 1363.477157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_dropped -r 2 [ 1363.508363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_dropped -r 2 [ 1363.541623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp 192.168.201.135@tcp [ 1363.567435] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1363.748311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1363.773725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1363.798594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1363.823467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1363.853461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1363.877636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1363.905775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1363.930746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1363.946431] LNet: 88142:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1363.973964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1364.010354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1364.039796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1364.063377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1364.091719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1364.117318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1364.411623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_aborted -r 2 [ 1364.435573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_aborted -r 2 [ 1364.458045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_aborted -r 2 [ 1364.482308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_aborted -r 2 [ 1364.505291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_aborted -r 2 [ 1364.531537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_aborted -r 2 [ 1364.564546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_aborted -r 2 [ 1364.595559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_aborted -r 2 [ 1364.633087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp 192.168.201.135@tcp [ 1364.651680] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1364.841707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1364.861617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1364.880644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1364.893321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1364.920105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1364.955324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1364.977568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1364.997568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1365.014590] LNet: 88262:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1365.035586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1365.073773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1365.107974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1365.135695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1365.161680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1365.200145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1365.530772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_no_route -r 2 [ 1365.567593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_no_route -r 2 [ 1365.596550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_no_route -r 2 [ 1365.626590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_no_route -r 2 [ 1365.656412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_no_route -r 2 [ 1365.687118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_no_route -r 2 [ 1365.707449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_no_route -r 2 [ 1365.724964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_no_route -r 2 [ 1365.740650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp 192.168.201.135@tcp [ 1365.897501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1365.917188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1365.943901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1365.968871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1365.989744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1366.010647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1366.032893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1366.051885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1366.079522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1366.104680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1366.134154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1366.157652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1366.184502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1366.206239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1366.526945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_timeout -r 2 [ 1366.548854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_timeout -r 2 [ 1366.574918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_timeout -r 2 [ 1366.600917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_timeout -r 2 [ 1366.627401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_timeout -r 2 [ 1366.660387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_timeout -r 2 [ 1366.694434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_timeout -r 2 [ 1366.719140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_timeout -r 2 [ 1366.751761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp 192.168.201.135@tcp [ 1366.763652] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1366.767676] LNet: Skipped 1 previous similar message [ 1366.893393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1366.928168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1366.952410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1366.974332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1366.992768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1367.016864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1367.040635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1367.065305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1367.074632] LNet: 88503:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1367.077997] LNet: 88503:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1367.088396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1367.112837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1367.140977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1367.160161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1367.179651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1367.195390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1367.560375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e remote_dropped -r 2 [ 1367.582936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e remote_dropped -r 2 [ 1367.603560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e remote_dropped -r 2 [ 1367.626192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e remote_dropped -r 2 [ 1367.649749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e remote_dropped -r 2 [ 1367.674252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e remote_dropped -r 2 [ 1367.693885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e remote_dropped -r 2 [ 1367.714704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e remote_dropped -r 2 [ 1367.736539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp 192.168.201.135@tcp [ 1367.750509] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.135@tcp [ 1367.892186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1367.913736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1367.936889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1367.965745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1367.991775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1368.010861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1368.038441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1368.059497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1368.080498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1368.106034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1368.135496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1368.167307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1368.200423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1368.229170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1370.256183] LNet: 88696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1370.260373] LNetError: 88696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.295301] LNet: Removed LNI 192.168.201.35@tcp [ 1371.297153] LNetError: 88696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.307586] LNet: Removed LNI 192.168.201.35@tcp1 [ 1371.511538] Key type .llcrypt unregistered [ 1371.513307] Key type ._llcrypt unregistered [ 1381.716304] Key type ._llcrypt registered [ 1381.717470] Key type .llcrypt registered [ 1382.721187] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 11:12:51 (1788361971) [ 1382.923984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1382.938906] alg: No test for adler32 (adler32-zlib) [ 1383.839684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1383.874227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1383.888704] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1384.534470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1384.563216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.35@tcp [ 1384.577927] LNetError: 89674:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.35@tcp to recovery [ 1384.669943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.35@tcp [ 1384.774677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1388.966856] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 11:12:58 (1788361978) [ 1389.014184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1389.045535] LNet: 90260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.055512] LNetError: 90260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.076694] LNet: Removed LNI 192.168.201.35@tcp [ 1389.098262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1389.133863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1389.151686] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1389.529761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1389.541253] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1390.119629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.35@tcp [ 1390.131570] LNetError: 90531:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.35@tcp to recovery [ 1390.176515] LNet: 1 local NIs in recovery (showing 1): 192.168.201.35@tcp [ 1390.180168] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1391.280373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.35@tcp [ 1396.015414] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 11:13:05 (1788361985) [ 1397.198322] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg135-server [ 1398.341073] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 11:13:07 (1788361987) [ 1399.519616] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg135-server [ 1400.650894] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 11:13:09 (1788361989) [ 1401.867812] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg135-server [ 1403.042946] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 11:13:12 (1788361992) [ 1403.275612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1403.298657] LNet: 92152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1403.307489] LNetError: 92152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1403.319355] LNet: Removed LNI 192.168.201.35@tcp [ 1403.330889] LNet: Removed LNI 192.168.201.35@tcp1 [ 1403.345129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1403.382936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1403.397425] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1403.794759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1403.832753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1403.853064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1403.902909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1403.934306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1403.968487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1408.026480] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 11:13:17 (1788361997) [ 1408.158666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1408.179900] LNet: 92877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1408.193801] LNetError: 92877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1408.201242] LNetError: 92877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1408.213850] LNet: Removed LNI 192.168.201.35@tcp [ 1408.243877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1408.293246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1408.309514] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1413.633298] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 11:13:22 (1788362002) [ 1413.704712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1413.725774] LNet: 93571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1413.732901] LNetError: 93571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1413.746401] LNet: Removed LNI 192.168.201.35@tcp [ 1413.782514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1413.845538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1413.876363] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1419.283460] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 11:13:28 (1788362008) [ 1420.520884] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1421.990526] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 11:13:30 (1788362010) [ 1422.321519] LNet: 94614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.329321] LNetError: 94614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1422.339403] LNet: Removed LNI 192.168.201.35@tcp [ 1422.805927] Key type .llcrypt unregistered [ 1422.808944] Key type ._llcrypt unregistered [ 1426.917278] Key type ._llcrypt registered [ 1426.919292] Key type .llcrypt registered [ 1427.098459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1427.112672] alg: No test for adler32 (adler32-zlib) [ 1428.028782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1428.053137] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1433.433583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1436.793406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.135@tcp [ 1437.088670] LNet: 95030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1437.092875] LNetError: 95030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1437.107234] LNet: Removed LNI 192.168.201.35@tcp [ 1437.447475] Key type .llcrypt unregistered [ 1437.452137] Key type ._llcrypt unregistered [ 1444.717535] Key type ._llcrypt registered [ 1444.719496] Key type .llcrypt registered [ 1445.505457] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1446.529141] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1447.588785] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 11:13:56 (1788362036) [ 1448.827179] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1449.950846] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 11:13:58 (1788362038) [ 1451.206468] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg135-server [ 1452.450982] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 11:14:01 (1788362041) [ 1453.804187] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg135-server [ 1455.100790] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 11:14:04 (1788362044) [ 1456.366825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1456.394846] alg: No test for adler32 (adler32-zlib) [ 1457.359916] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 1457.523428] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1459.183409] Key type lgssc registered [ 1460.233475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1474.279463] Key type lgssc unregistered [ 1474.480854] LNet: 98202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.485135] LNetError: 98202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1474.497202] LNet: Removed LNI 192.168.201.35@tcp [ 1474.984643] Key type .llcrypt unregistered [ 1474.986677] Key type ._llcrypt unregistered [ 1485.882482] Key type ._llcrypt registered [ 1485.884620] Key type .llcrypt registered [ 1486.891397] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 11:14:35 (1788362075) [ 1487.272443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1487.292974] alg: No test for adler32 (adler32-zlib) [ 1488.183209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1488.229240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1488.261312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1488.296885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1488.334580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1493.399562] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 11:14:42 (1788362082) [ 1493.583145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1493.595951] LNet: 99617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.608904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1493.642685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1493.655677] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1494.259463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1494.272642] LNet: 99761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.275627] LNetError: 99761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.286716] LNet: Removed LNI 192.168.201.35@tcp [ 1494.300025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.335927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1494.360972] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1494.724193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1494.760454] LNet: Removed LNI 192.168.201.35@tcp [ 1494.776972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.814521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1494.843098] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1499.963935] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 11:14:48 (1788362088) [ 1500.051168] LNet: 100373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.055272] LNet: 100373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1500.062564] LNetError: 100373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.067122] LNetError: 100373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1500.085288] LNet: Removed LNI 192.168.201.35@tcp [ 1500.509864] Key type .llcrypt unregistered [ 1500.512657] Key type ._llcrypt unregistered [ 1505.212226] Key type ._llcrypt registered [ 1505.214439] Key type .llcrypt registered [ 1505.400387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1505.410727] alg: No test for adler32 (adler32-zlib) [ 1506.363325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1506.436933] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1506.440789] LNet: *** cfs_fail_loc=e003, val=100*** [ 1506.946412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1506.958106] LNet: *** cfs_fail_loc=e003, val=100*** [ 1506.959938] LNet: Skipped 2 previous similar messages [ 1506.964844] Ping reply received with mapping: [ 1506.966996] found metadata reply NI[192.168.201.35@tcp] [ 1506.969883] 23 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1506.972556] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1506.975602] 5a 5a 5a 5a 5a 5a 5a 5a [ 1507.044570] LNet: 100776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1507.049294] LNetError: 100776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1507.059806] LNet: Removed LNI 192.168.201.35@tcp [ 1507.429363] Key type .llcrypt unregistered [ 1507.432563] Key type ._llcrypt unregistered [ 1515.924353] Key type ._llcrypt registered [ 1515.926373] Key type .llcrypt registered [ 1516.856581] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 11:15:05 (1788362105) [ 1517.410542] Key type .llcrypt unregistered [ 1517.417476] Key type ._llcrypt unregistered [ 1522.418438] Key type ._llcrypt registered [ 1522.420268] Key type .llcrypt registered [ 1522.572181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1522.582940] alg: No test for adler32 (adler32-zlib) [ 1523.591796] LNetError: 101820:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1523.837388] Key type .llcrypt unregistered [ 1523.839029] Key type ._llcrypt unregistered [ 1527.994445] Key type ._llcrypt registered [ 1527.995776] Key type .llcrypt registered [ 1528.964139] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 11:15:17 (1788362117) [ 1529.473356] Key type .llcrypt unregistered [ 1529.475702] Key type ._llcrypt unregistered [ 1534.541933] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1534.547802] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1537.010588] Key type ._llcrypt registered [ 1537.015433] Key type .llcrypt registered [ 1537.169082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1537.179366] alg: No test for adler32 (adler32-zlib) [ 1538.036880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.069082] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1546.166569] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 11:15:35 (1788362135) [ 1546.256580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.269612] LNet: 103597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.272597] LNetError: 103597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1546.290159] LNet: Removed LNI 10.1.2.3@tcp [ 1546.312722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.927897] Key type .llcrypt unregistered [ 1546.929504] Key type ._llcrypt unregistered [ 1551.170002] Key type ._llcrypt registered [ 1551.171176] Key type .llcrypt registered [ 1551.329371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1551.340672] alg: No test for adler32 (adler32-zlib) [ 1552.226818] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1552.279963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1552.392506] LNet: 103889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.399281] LNetError: 103889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1552.415645] LNet: Removed LNI 192.168.201.35@tcp [ 1552.861632] Key type .llcrypt unregistered [ 1552.863119] Key type ._llcrypt unregistered [ 1560.817957] Key type ._llcrypt registered [ 1560.819965] Key type .llcrypt registered [ 1561.645374] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 11:15:50 (1788362150) [ 1562.158635] Key type .llcrypt unregistered [ 1562.161182] Key type ._llcrypt unregistered [ 1566.102750] Key type ._llcrypt registered [ 1566.104804] Key type .llcrypt registered [ 1566.221190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1566.232697] alg: No test for adler32 (adler32-zlib) [ 1571.043436] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 11:16:00 (1788362160) [ 1571.107564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1571.135327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.174371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1571.200600] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1571.696791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1571.727145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1571.750650] LNet: Removed LNI 192.168.201.35@tcp [ 1571.902077] LNet: 105631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.162643] Key type .llcrypt unregistered [ 1572.165632] Key type ._llcrypt unregistered [ 1578.927095] Key type ._llcrypt registered [ 1578.929016] Key type .llcrypt registered [ 1579.696420] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 11:16:08 (1788362168) [ 1579.966618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1579.978350] alg: No test for adler32 (adler32-zlib) [ 1580.834667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.880597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1580.901641] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1581.319901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1581.342952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.35@tcp [ 1581.355875] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1581.358828] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1581.362755] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1581, 0, 0 [ 1581.370370] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1581.372377] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1581.380515] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1581.384733] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1581, 0, 0 [ 1581.388800] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1581.391150] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1581.393880] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1581.396382] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1581, 0, 1631 [ 1581.399381] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1581.401477] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1581.404456] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1581.406815] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1581, 0, 1631 [ 1581.410795] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1585.075681] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 11:16:14 (1788362174) [ 1585.388309] LNet: 107170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.391517] LNetError: 107170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1585.408129] LNet: Removed LNI 192.168.201.35@tcp [ 1585.852494] Key type .llcrypt unregistered [ 1585.854435] Key type ._llcrypt unregistered [ 1589.806335] Key type ._llcrypt registered [ 1589.810980] Key type .llcrypt registered [ 1589.974265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1589.995824] alg: No test for adler32 (adler32-zlib) [ 1590.830435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1590.845557] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1595.588924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1601.581823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.617625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1601.630864] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1602.507891] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1602.579551] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1602.584883] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1602.587516] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1604.875424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1604.897927] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1611.373236] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 11:16:40 (1788362200) [ 1611.498811] LNet: 108679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1611.503243] LNetError: 108679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1611.511866] LNet: Removed LNI 192.168.201.35@tcp [ 1612.575174] LNet: Removed LNI 192.168.201.35@tcp1 [ 1612.581745] LNetError: 108679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1612.591228] LNetError: 108679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1612.610269] LNet: Removed LNI 10.1.2.3@tcp99 [ 1613.063182] Key type .llcrypt unregistered [ 1613.064660] Key type ._llcrypt unregistered [ 1617.031404] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1617.038568] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1619.404338] Key type ._llcrypt registered [ 1619.406616] Key type .llcrypt registered [ 1619.515968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1619.527231] alg: No test for adler32 (adler32-zlib) [ 1620.366576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.400874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1620.419610] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1620.861795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1620.874890] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1622.116497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1622.147108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1622.178642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1622.198779] LNetError: 109423:0:(peer.c:2277:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1622.222205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1622.245421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1622.504492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1622.559427] LNet: 109443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.568554] LNetError: 109443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1622.576520] LNet: Removed LNI 10.1.2.3@tcp [ 1623.589785] LNet: Removed LNI 192.168.201.35@tcp [ 1623.985749] Key type .llcrypt unregistered [ 1623.987733] Key type ._llcrypt unregistered [ 1630.508140] Key type ._llcrypt registered [ 1630.510049] Key type .llcrypt registered [ 1631.379971] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 11:17:00 (1788362220) [ 1631.684371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1631.696627] alg: No test for adler32 (adler32-zlib) [ 1632.564194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.607686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1632.629969] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1633.048111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1633.088042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg135-client@tcp [ 1636.982913] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 11:17:05 (1788362225) [ 1637.092868] LNet: 110973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1637.098359] LNetError: 110973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1638.121333] LNet: Removed LNI 192.168.201.35@tcp [ 1638.649404] Key type .llcrypt unregistered [ 1638.650824] Key type ._llcrypt unregistered [ 1642.379948] Key type ._llcrypt registered [ 1642.383374] Key type .llcrypt registered [ 1642.498368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1642.512895] alg: No test for adler32 (adler32-zlib) [ 1643.381768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1643.398174] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1643.413464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1646.900559] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 11:17:16 (1788362236) [ 1647.901867] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1649.103773] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 11:17:18 (1788362238) [ 1649.973970] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1651.047187] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 11:17:20 (1788362240) [ 1651.127371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1651.163585] LNet: 112388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1651.172133] LNetError: 112388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1651.192081] LNet: Removed LNI 192.168.201.35@tcp [ 1651.221507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.275758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1651.319329] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1651.957315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1651.996558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1656.284342] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 11:17:25 (1788362245) [ 1656.328741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.346125] LNet: 113098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1656.352907] LNetError: 113098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1656.368077] LNet: Removed LNI 192.168.201.35@tcp [ 1656.412435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.459973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1656.490937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1656.818431] Key type .llcrypt unregistered [ 1656.819939] Key type ._llcrypt unregistered [ 1664.052149] Key type ._llcrypt registered [ 1664.054437] Key type .llcrypt registered [ 1665.092178] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 11:17:33 (1788362253) [ 1665.366166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1665.377337] alg: No test for adler32 (adler32-zlib) [ 1666.287977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.346817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1666.368945] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1666.853927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1666.901518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1671.307890] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 11:17:40 (1788362260) [ 1671.566048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1671.583276] LNet: 114658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1671.585803] LNetError: 114658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1671.595348] LNet: Removed LNI 192.168.201.35@tcp [ 1671.626277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1671.663315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1671.693472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1675.878336] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 11:17:44 (1788362264) [ 1675.967584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.992687] LNet: 115233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.020070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.104506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1676.317653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1676.345991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1676.382359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1676.598123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1676.620718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1676.673584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1676.697278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.739992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1677.008463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1677.032941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1677.084428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1677.423707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1677.457578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1682.163507] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 11:17:51 (1788362271) [ 1682.291880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1682.327061] LNet: 115867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1682.329983] LNet: 115867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1682.363990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.403234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1682.420166] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1682.935246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1682.951830] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 1683.585924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1683.633504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1683.809765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.822801] LNetError: 116150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.826939] LNet: Removed LNI 192.168.201.35@tcp [ 1683.843864] LNet: Removed LNI 192.168.201.35@tcp2 [ 1683.866650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.918195] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1683.922159] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 1683.925888] LNet: 116156:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1684.305100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1684.460622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.476347] LNet: 116194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.480156] LNet: 116194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1685.535198] LNet: Removed LNI 192.168.201.35@tcp [ 1685.538225] LNetError: 116194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.543855] LNetError: 116194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1685.557982] LNet: Removed LNI 192.168.201.35@tcp2 [ 1685.574342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.615797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1685.631485] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1686.199152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1686.215237] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 1686.979645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1687.015841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1687.059116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1687.226963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.287342] LNet: Removed LNI 192.168.201.35@tcp [ 1688.289362] LNetError: 116478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.293179] LNetError: 116478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1688.308738] LNet: Removed LNI 192.168.201.35@tcp2 [ 1688.324470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.379155] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1688.388758] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 1688.395952] LNet: 116484:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1688.730224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1688.868900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.885827] LNet: 116521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.891209] LNet: 116521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1688.897704] LNet: Removed LNI 192.168.201.35@tcp [ 1689.958309] LNet: Removed LNI 192.168.201.35@tcp2 [ 1689.980706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.024859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1690.043432] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1690.521627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1690.537057] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 1691.073904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1691.113848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1691.261543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1692.319647] LNet: Removed LNI 192.168.201.35@tcp [ 1692.326078] LNetError: 116801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1692.331610] LNetError: 116801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1692.343508] LNet: Removed LNI 192.168.201.35@tcp2 [ 1692.373606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.422583] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1692.429933] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 1692.434030] LNet: 116807:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1692.842428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1692.960088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1693.986150] LNet: Removed LNI 192.168.201.35@tcp [ 1693.999415] LNet: Removed LNI 192.168.201.35@tcp2 [ 1694.025213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.091717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1694.115287] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1694.641186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1694.662928] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 1695.252451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1695.272765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1695.305034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1695.444992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.479328] LNet: Removed LNI 192.168.201.35@tcp [ 1697.509746] LNet: Removed LNI 192.168.201.35@tcp2 [ 1697.555517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.608090] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1697.613622] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 1697.618928] LNet: 117135:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1697.940920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1698.186463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.204852] LNet: 117172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1698.209029] LNet: 117172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1698.214462] LNet: Removed LNI 192.168.201.35@tcp [ 1699.237815] LNet: Removed LNI 192.168.201.35@tcp2 [ 1699.283086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.332921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1699.360283] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1699.901745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1699.916736] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 1700.535432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1700.553781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1700.588675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1700.817035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.840798] LNetError: 117469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.843746] LNetError: 117469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1700.847850] LNet: Removed LNI 192.168.201.35@tcp [ 1700.858172] LNet: Removed LNI 192.168.201.35@tcp2 [ 1700.870902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.917458] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1700.923437] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 1701.255283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1701.433910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.495367] LNet: Removed LNI 192.168.201.35@tcp [ 1702.513209] LNet: Removed LNI 192.168.201.35@tcp2 [ 1702.541344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.581372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1702.597981] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1703.042788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1703.050895] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 1703.511546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1703.538366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1703.563190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1703.603493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1703.782631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.799287] LNet: Removed LNI 192.168.201.35@tcp [ 1705.834468] LNet: Removed LNI 192.168.201.35@tcp2 [ 1705.866348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.931384] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1705.939313] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 1705.942797] LNet: 117835:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1705.946070] LNet: 117835:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1706.378835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1711.453242] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1712.493456] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 11:18:21 (1788362301) [ 1712.620971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.631231] LNet: Removed LNI 192.168.201.35@tcp [ 1713.701934] LNet: Removed LNI 192.168.201.35@tcp2 [ 1713.728386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.983385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1714.007281] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1714.464597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1714.482453] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1715.114952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.132522] LNet: 118868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1715.141593] LNet: 118868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1715.150660] LNet: Removed LNI 10.1.2.3@tcp [ 1715.161327] LNet: Removed LNI 192.168.201.35@tcp [ 1715.182732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.245906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1715.286444] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1715.293620] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1715.642557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1715.660366] LNet: Removed LNI 192.168.201.35@tcp [ 1715.674652] LNet: Removed LNI 10.1.2.3@tcp [ 1720.254452] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 11:18:29 (1788362309) [ 1720.413880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.439940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.503280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1720.545049] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1721.082660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1721.116978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1721.144734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1721.173554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1721.198096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1721.223316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1721.254211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1721.285970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1721.320492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1721.354733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1721.373254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1721.520450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1726.421940] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 11:18:35 (1788362315) [ 1726.625908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.642304] LNetError: 120208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.647825] LNetError: 120208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 1726.662856] LNet: Removed LNI 192.168.201.35@tcp [ 1726.684797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.793965] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1726.943563] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1726.958802] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1726.966060] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1729.228575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1729.271374] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1729.896491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1729.937341] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1731.403839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1731.434748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1731.490902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.35@tcp -m PUT -e local_timeout -r 1 [ 1731.539747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1731.590281] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1731.598349] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1731.788627] LNetError: 120273:0:(lib-move.c:4801:lnet_parse()) 192.168.201.35@tcp, src 192.168.201.35@tcp: Dropping PUT (error -108 looking up sender) [ 1732.842936] LNet: Removed LNI 10.1.2.3@tcp [ 1733.861878] LNet: Removed LNI 192.168.201.35@tcp [ 1734.316709] Key type .llcrypt unregistered [ 1734.319519] Key type ._llcrypt unregistered [ 1742.321755] Key type ._llcrypt registered [ 1742.323627] Key type .llcrypt registered [ 1743.190565] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 11:18:52 (1788362332) [ 1743.562126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1743.581751] alg: No test for adler32 (adler32-zlib) [ 1744.464905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.739430] LNet: 121583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.129887] Key type .llcrypt unregistered [ 1745.132645] Key type ._llcrypt unregistered [ 1749.465530] Key type ._llcrypt registered [ 1749.467484] Key type .llcrypt registered [ 1749.583170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1749.593279] alg: No test for adler32 (adler32-zlib) [ 1750.443751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1750.464721] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1755.846735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1755.978040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1756.363238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.135@tcp [ 1756.523618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1756.541454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1756.557346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1756.574835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1756.593122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1756.612056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1756.635360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1756.642703] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.135@tcp [ 1756.666420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1756.678735] LNet: 122033:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1756.696476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1756.727479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1756.763722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1756.794982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1756.814279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1756.837451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1757.269752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1757.393126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1757.411324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1757.430591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1757.448771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1757.478361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1757.507286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1757.534146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1757.550959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1757.558726] LNet: 122150:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1757.570962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1757.609212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1757.644846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1757.680659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1757.698669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1757.718457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1758.089617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.135@tcp [ 1758.212511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1758.235382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1758.254944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1758.272027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1758.293532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1758.315263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1758.333751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1758.357315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1758.376537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1758.395868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1758.411934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1758.432521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1758.456662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1758.482862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1758.872814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1758.997660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1759.011787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1759.026741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1759.055081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1759.090442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1759.118825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1759.142145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1759.170700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1759.184580] LNet: 122384:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1759.189717] LNet: 122384:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1759.205548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1759.234274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1759.269972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1759.297760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1759.322805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1759.345807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1759.518839] LNet: 122430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.522816] LNetError: 122430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.537894] LNet: Removed LNI 192.168.201.35@tcp [ 1759.781698] Key type .llcrypt unregistered [ 1759.783790] Key type ._llcrypt unregistered [ 1770.548716] Key type ._llcrypt registered [ 1770.551236] Key type .llcrypt registered [ 1771.497810] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 11:19:20 (1788362360) [ 1771.804675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1771.819705] alg: No test for adler32 (adler32-zlib) [ 1772.679120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.913551] LNet: 123317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1773.349461] Key type .llcrypt unregistered [ 1773.355189] Key type ._llcrypt unregistered [ 1776.938847] Key type ._llcrypt registered [ 1776.940147] Key type .llcrypt registered [ 1777.096683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1777.107888] alg: No test for adler32 (adler32-zlib) [ 1777.996140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1778.022165] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1783.015325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1787.482988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1787.499791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1787.512590] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1787.984591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1788.323761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.135@tcp [ 1788.451528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1788.478293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1788.497163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1788.518931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1788.542419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1788.559978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1788.580563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1788.600668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1788.612408] LNet: 123968:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1788.623949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1788.644070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1788.658506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1788.676963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1788.697463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1788.714125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1789.105553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1789.255335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1789.278428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1789.283422] LNet: 2 peer NIs in recovery (showing 2): 192.168.201.135@tcp, 192.168.201.135@tcp1 [ 1789.299567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1789.324871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1789.347775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1789.368469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1789.386336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1789.411736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1789.434190] LNet: 124085:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1789.455686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1789.479887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1789.503936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1789.526399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1789.551237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1789.575892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1789.982668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.135@tcp [ 1790.150246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1790.175568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1790.199356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1790.220375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1790.245559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1790.274858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1790.302677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1790.325993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1790.345278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1790.366683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1790.387929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1790.406692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1790.434906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1790.461243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1790.819691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1790.949818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1790.972791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1790.990356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1791.006609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1791.024875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1791.044037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1791.066442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1791.085886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1791.100401] LNet: 124319:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1791.109023] LNet: 124319:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1791.121207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1791.141512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1791.165058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1791.189884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1791.204853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1791.222212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1794.266964] LNet: 124400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1794.270814] LNetError: 124400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1794.278345] LNet: Removed LNI 192.168.201.35@tcp [ 1794.292829] LNet: Removed LNI 192.168.201.35@tcp1 [ 1794.541758] Key type .llcrypt unregistered [ 1794.543370] Key type ._llcrypt unregistered [ 1804.636170] Key type ._llcrypt registered [ 1804.638853] Key type .llcrypt registered [ 1805.576962] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 11:19:54 (1788362394) [ 1806.502310] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 1807.561539] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 11:19:56 (1788362396) [ 1811.746960] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 11:20:00 (1788362400) [ 1816.448889] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 11:20:05 (1788362405) [ 1821.571868] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 11:20:10 (1788362410) [ 1826.469481] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 11:20:15 (1788362415) [ 1832.089293] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 11:20:21 (1788362421) [ 1837.708892] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 11:20:26 (1788362426) [ 1842.829274] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 11:20:31 (1788362431) [ 1848.102419] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 11:20:36 (1788362436) [ 1852.868985] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 11:20:41 (1788362441) [ 1857.387861] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 11:20:46 (1788362446) [ 1861.683548] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 11:20:50 (1788362450) [ 1865.948291] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 11:20:55 (1788362455) [ 1870.157119] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 11:20:59 (1788362459) [ 1874.423959] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 11:21:03 (1788362463) [ 1879.239529] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 11:21:08 (1788362468) [ 1883.650868] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 11:21:12 (1788362472) [ 1888.406984] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 11:21:17 (1788362477) [ 1888.708971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1888.733544] alg: No test for adler32 (adler32-zlib) [ 1889.659632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1889.746319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1889.764202] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1890.550918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1890.562877] LNet: 135394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1890.566611] LNetError: 135394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1890.583575] LNet: Removed LNI 192.168.201.35@tcp [ 1890.597294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1890.656522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1890.679566] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1895.057926] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 11:21:24 (1788362484) [ 1899.356686] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 11:21:28 (1788362488) [ 1903.716038] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 11:21:32 (1788362492) [ 1908.028505] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 11:21:36 (1788362496) [ 1912.184048] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 11:21:41 (1788362501) [ 1916.770225] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 11:21:45 (1788362505) [ 1921.875418] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 11:21:50 (1788362510) [ 1926.399632] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 11:21:55 (1788362515) [ 1930.619065] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 11:21:59 (1788362519) [ 1934.303428] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 11:22:03 (1788362523) [ 1938.019409] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 11:22:07 (1788362527) [ 1941.784618] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 11:22:10 (1788362530) [ 1945.655417] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1679 sec ========== 11:22:14 (1788362534) [ 1945.796125] LNet: 142852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.798839] LNetError: 142852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.812536] LNet: Removed LNI 192.168.201.35@tcp [ 1946.286757] Key type .llcrypt unregistered [ 1946.290615] Key type ._llcrypt unregistered [ 1952.280525] Key type ._llcrypt registered [ 1952.283190] Key type .llcrypt registered [ 1952.523462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1952.532865] alg: No test for adler32 (adler32-zlib) [ 1953.490196] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 1953.655035] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1955.271173] Key type lgssc registered [ 1955.919136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1992.045996] Lustre: Mounted lustre-client [ 1996.030542] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2000.857970] Lustre: Unmounted lustre-client [ 2030.093546] Key type lgssc unregistered [ 2030.268789] LNet: 145775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2030.273362] LNetError: 145775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2030.282952] LNet: Removed LNI 192.168.201.35@tcp [ 2030.691528] Key type .llcrypt unregistered [ 2030.693489] Key type ._llcrypt unregistered